Create a `./tmp` with `memfs` and have the rest of the file system still working

[ChrisCinelli]

Last night I was quite tired for lack of sleeping but I had been tinkering for a few hours with patchFs, requireFs, memfs, unionfs, linkfs. I went through a few errors with short stacktraces that took time to investigate.

The first attempt was just trying to use memfs. requires do not work anymore:

const fsMonkey = require('fs-monkey');
const vol = require('memfs').vol;
fsMonkey.patchFs(vol); //Tried patchRequire too

Realized that the goal was to create ./tmp with memfs: 1) Attempt

const fsMonkey = require('fs-monkey');
const fs = require('fs');
const memfs = require('memfs').Volume.fromJSON({'./tmp/test.txt': 'dummy content '});
const ufs = require('unionfs').ufs;
ufs
  .use(fs) // if you switch the 2 `.use` you get different errors 
  .use(memfs)
fsMonkey.patchFs(ufs);

2) Attempt:

const fsMonkey = require('fs-monkey');
const fs = require('fs');
const lfs = require('linkfs').link(fs, ['./tmp', 'tmpfs'])
const memfs = require('memfs').Volume.fromJSON({'tmpfs/test.txt': 'dummy content'});
const ufs = require('unionfs').ufs;
ufs
  .use(memfs)
  .use(lfs)
fsMonkey.patchFs(ufs);

3) Attempt:

const fsMonkey = require('fs-monkey');
const fs = require('fs');
const memfs = require('memfs').Volume.fromJSON({'/tmpfs/test.txt': 'dummy content'});
const ufs = require('unionfs').ufs;
ufs
  .use(memfs)
  .use(fs)
const lfs = require('linkfs').link(ufs, ['./tmp', '/tmpfs'])

fsMonkey.patchFs(lfs);

How do you combine patchFs, requireFs, memfs, unionfs, linkfs so fs' calls everywhere in the app work seamlessly ?

I am sure if I try to solve this when I am rested I could make it quickly work. At the same time, the documentation could be a little more easy to find and cohesive. See: https://github.com/streamich/memfs/issues/292

[Sly1024]

@ChrisCinelli I think I know what you were trying to do and I had the same issues, but I got it working at the end.

My goal was to create a /virtual folder and merge it with the "real" filesystem and then patch it back to "fs" so if any code tries to read/write files then it works both on real files and virtual files.

I looked at the code of unionfs and basically it has an array of filesystems and when you try to do anything (e.g. readFileSync) it loops through each fs and tries to call that function. It stops at the first fs that succeeds. When you read a file, it's usually obvious that it's either on real fs or memfs, but what happens when you try to write a file? It turns out that you can't write "./local-file" to memfs, it only works with "/absolute/paths". So I used memfs first then the real fs. This created a "ufs" object that worked as I wanted.

ufs.use(memfs).use(fs);

But I also wanted to patch it back to the real "fs" object, so other libraries would be able to read/write virtual files. Easy, right? Just: patchFs(ufs). And it didn't work.

I found out the reason. If you look at the line above: ufs.use(memfs).use(fs); - this means: try memfs first, then fs. But if you patch ufs's functions back to 'fs' then it really becomes: try memfs first then ufs. I'm not sure why it did not go into an infinite recursion to be honest, but I fixed the issue by backing up the original "fs" object and use()-ing that instead of "fs".

Here's the full test code:

const fs = require('fs');
const { ufs } = require('unionfs');
const { fs:memfs, vol } = require('memfs');
const { patchFs } = require('fs-monkey');

//back up original 'fs'
const ofs = { ...fs };

vol.fromJSON({
    './dummy.txt': 'Hello World'    // without at least 1 entry, it doesn't work - why??
}, '/virtual');

ufs.use(vol).use(ofs); // <-- this is the crucial part!
patchFs(ufs);

fs.writeFileSync('./hello.txt', 'Are you there?');  // This writes a real file in the current dir
fs.writeFileSync('/virtual/hello.txt', 'You\'re in the matrix!');   // this writes a virtual file into memory

// we can read back both with "fs":
console.log(fs.readFileSync('./hello.txt', 'utf8'));
console.log(fs.readFileSync('/virtual/hello.txt', 'utf8'));

[WinstonN]

This issue is incredibly important. I count myself lucky to have found it. @Sly1024 thank you for sharing this. I'm still not sure how you figured it out (or how it's working tbh) but it works :P

[G-Rath]

It turns out that you can't write "./local-file" to memfs, it only works with "/absolute/paths

You can write relative paths to memfs, it just won't create any paths for you, meaning that process.cwd() has to be created first :)

This is how I setup my file system for mocking in jest tests for my cli project:

import * as fs from 'fs';
import { createFsFromVolume, vol } from 'memfs';
import { ufs } from 'unionfs';

/**
 * Factory that provides the real file-system union-d with an in-memory one.
 *
 * Files that don't exist in the in-memory FS will be read from the underlying
 * real file system, while any writes will take place on the in-memory FS>
 *
 * @returns {typeof fs}
 */
const mockFsFactory = (): typeof fs => {
  const fs = jest.requireActual('fs');

  beforeEach(() => vol.mkdirSync(process.cwd(), { recursive: true }));
  afterEach(() => vol.reset());

  return ufs.use(fs).use(createFsFromVolume(vol) as typeof fs);
};

export = mockFsFactory;

[Shkeats]

@Sly1024 thanks so much for writing that up it was really helpful. I was thoroughly confused trying to achieve the same usecase.

[radist2s]

Vitest & memfs example:

import { vi } from 'vitest';

const createMemFs = vi.hoisted(
  () => async (fsOriginal: typeof import('node:fs')) => {
    const { Volume, createFsFromVolume } = await import('memfs');

    return createFsFromVolume(
      Volume.fromJSON({
        '/memfs/myFile': fsOriginal.readFileSync('/myFile', 'utf-8'),
      })
    );
  }
);

vi.mock('node:fs', async (importOriginal) => {
  return {
    default: await createMemFs(
      await importOriginal<typeof import('node:fs')>()
    ),
  };
});

vi.mock('fs', async (importOriginal) => {
  return {
    default: await createMemFs(
      await importOriginal<typeof import('fs')>()
    ),
  };
});