Integration with the Emscripten file system in a webworker

[oeway]

Hi, I am trying to integrate the emscripten file system work with BrowserFS in a webworker. Here is the setup I want to achieve:

1. in the main thread, mount a indexeddb file system with browserfs
2. in the worker thread I would like to attach to the file system in the main thread, create an emscripten interface for browserfs and mount it to a WASM library.

As I understand from a comment here, I would also need the AsyncMirror to make an sync interface for the emscripten interface.

This is what I have tried but failed to get it work:

Main thread

  // Listen for remote file system requests.
  BrowserFS.FileSystem.WorkerFS.attachRemoteListener(webWorkerObject);

WebWorker

BrowserFS.configure({
  fs: "AsyncMirror",
  options: {
    sync: { fs: "InMemory" },
    async: {  fs: "WorkerFS", options: { worker: self } }
  }
}, function(e) {
  if (e) {
    // An error happened!
    throw e;
  }
  let FS = my_wasm_module.FS;
  let PATH = my_wasm_module.PATH;
  // Create an Emscripten interface for the BrowserFS
  const BFS = new BrowserFS.EmscriptenFS(FS, PATH);
  // Create mount point in Emscripten FS
  FS.createFolder(FS.root, 'data', true, true);
  // Mount BrowserFS into Emscripten FS
  FS.mount(BFS, {root: '/'}, '/data');
});

Is this the way to use it?

[cloudwalkerfre]

@oeway Hey, are you trying to bind it with pyodide? Was doing the same thing here, hope you find a solution, running pyodide in main thread is causing my App render to suffer, I guess this is the only way to do it.

[oeway]

Yes, I am trying to use it with Pyodide, however I haven't figure it out yet.

[cloudwalkerfre]

I was able to do just the normal WorkerFS.Create, then have the cb fs as global, call it in pyodide like

import js
import pandas as pd

js.fs.mkdir('test')

I can see the newly created folder in mainthread, I guess we can take a detour, something like pd.read_csv(StringIO(js.fs.readFile)), but that's not pretty

[leviathan747]

@oeway I was able to get it working with the following in the web worker:

// set up file system
await new Promise((resolve, reject) => {
  BrowserFS.configure({
    fs: 'AsyncMirror',
    options: {
      sync: { fs: 'InMemory' },
      async: { fs: 'WorkerFS', options: { worker: self } }
    }   
  }, function(e) {
    if (e) { reject(e); }
    resolve();
  }); 
});
const BFS = new BrowserFS.EmscriptenFS(Module.FS, Module.PATH, Module.ERRNO_CODES);
Module.FS.mkdir('/data');
Module.FS.mount(BFS, {root: '/'}, '/data');

I used your code as a base, but the part with ERRNO_CODES ended up being key for me -- I was getting errors anytime I tried to stat a file.