Is it possible to use the IndexedDB backend in a worker ?

[robert-westenberger]

In my main thread, I can configure zenfs to access a remote filesystem running in a webworker in a different context.

When using the InMemory backend, everything works as expected.

// worker.ts
import { attachFS, resolveMountConfig, InMemory } from '@zenfs/core';

const myFS = await resolveMountConfig({ backend: InMemory, name: 'foo' });
attachFS(self, myFS);

// main.ts
import { configure, Port, fs } from '@zenfs/core';
const worker = new Worker(new URL('./worker.ts', import.meta.url), {
      type: 'module',
    });

await configure({
  mounts: {
    '/port': {
      backend: Port,
      port: worker,
    },
  },
});

// Can now use methods on fs to interact with filesystem running in web worker thread..

But if I try to use the IndexedDB backend in the worker, i get an error.

// worker.ts
import { attachFS, resolveMountConfig } from '@zenfs/core';
import {IndexedDB} from "@zenfs/dom";

const myFS = await resolveMountConfig({ backend: IndexedDB, name: 'foo' });
attachFS(self, myFS);

The error is Error: RPC Failed,this line: https://github.com/zen-fs/core/blob/a6287a9bed0aea12b6259084d99efeccd9577459/src/backends/port/rpc.ts#L89

I have a feeling that I am not using the Port backend as its intended, apologies in advance if that is the case!

[james-pre]

@robert-westenberger Your code looks to be correct. Perhaps something in the worker fails? I suggest you check that indexedDB is available. If it is, then I'm not entirely sure what would cause a failure in the worker. Feel free to reach out if you have any questions.

[UnCor3]

@robert-westenberger Put your worker code in try catch block , in my case isomorphic-git was erroring out here is how i got it working without isomorphic-git

//worker.ts
import { attachFS, resolveMountConfig, fs } from "@zenfs/core";
import { IndexedDB } from "@zenfs/dom";

try {
    const myFS = await resolveMountConfig({ backend: IndexedDB });
    attachFS(self as any, myFS);

    //passed an empty object
    myFS.mkdirSync("/mnt", 1, {});

    //didn't work
    // await git.clone({
    //   fs: myFS as any,
    //   http,
    //   dir: "/",
    //   url: "https://github.com/isomorphic-git/lightning-fs",
    //   corsProxy: "https://cors.isomorphic-git.org",
    // });
} catch (error) {
    console.error(error);
}

//main.ts

import { configure, Port, fs } from "@zenfs/core";
import useWorkerStore from "@/store/worker.store";
export class GitWorkerIPC {
  public workerStore = useWorkerStore();
  public constructor() {
    this.init();
  }
  public async init() {
    try {
      await configure({
        mounts: {
          "/port": {
            backend: Port,
            //GitWorker is a Worker instance
            port: this.workerStore.GitWorker,
          },
        },
      });
        console.log(fs.readdirSync("/port")); //logs ["mnt"]
    } catch (error) {
        console.error(error);
    }
  }
}

@james-pre This is what fails to execute in isomorphic-git , most likely fs returned from resolveMountConfig misses properties ,its working fine if i do not use the Port backend and instead use Indexeddb backend on worker thread

Is Port backend fs missing some methods ? @james-pre see it in isomorphic-git source code

function bindFs(target, fs) {
  if (isPromiseFs(fs)) {
    for (const command of commands) {
    //TypeError: fs[command] is undefined
      target[`_${command}`] = fs[command].bind(fs);
    }
  } else {
    for (const command of commands) {
      target[`_${command}`] = pify(fs[command].bind(fs));
    }
  }

[james-pre]

@UnCor3,

@james-pre This is what fails to execute in isomorphic-git , most likely fs returned from resolveMountConfig misses properties ,its working fine if i do not use the Port backend and instead use Indexeddb backend on worker thread

If you look at the Mounting and unmounting, creating backends section of the readme, you will find a very clear notice:

[!WARNING] Instances of backends follow the internal ZenFS API. You should never use a backend's methods unless you are extending a backend.

I think this should answer your question. The internal API (FileSystem) is very different from the node:fs API. You should not be passing the result of resolveMountConfig to 3rd party APIs that expect a node:fs object. Instead, you should pass the fs exported by @zenfs/core. For example:

//worker.ts
import { attachFS, resolveMountConfig, fs } from '@zenfs/core';
import { IndexedDB } from '@zenfs/dom';

try {
    const myFS = await resolveMountConfig({ backend: IndexedDB });
    attachFS(self, myFS); // If a type cast is needed here please open another issue

    await git.clone({
        fs, // notice we are using the named `fs` export
        http, // assuming you've imported this
        dir: '/',
        url: 'https://github.com/isomorphic-git/lightning-fs',
        corsProxy: 'https://cors.isomorphic-git.org/',
    });
} catch (error) {
    console.error(error);
}

However, the IndexedDB backend is not mounted. I've found mounting a backend and using it with a Port sometimes does not work, so mounting it on the worker's fs may cause issues.

My suggestion for fixing your problem:

1. Use isomorphic-git from the main thread, which has the fs with a mounted Port, or
2. Mount the backend on the fs in the worker

Example worker for #​2:

    // ...
    const idbfs = await resolveMountConfig({ backend: IndexedDB });
    attachFS(self, idbfs); // If a type cast is needed here please open another issue
    fs.umount('/'); // unmounting the default in-memory mount on /
    fs.mount('/', idbfs);
    // ...

or

    // ...
    await configure({
        mounts: {
            /* Note: this is a convenience thing, `{ backend: IndexedDB }` works too.
            You can't pass options in this form */
            '/': IndexedDB 
        }
    });
    attachFS(self, fs.mounts.get('/')); // If a type cast is needed here please open another issue
    // ...

[robert-westenberger]

@james-pre When I console.log self.indexedDB in the worker, it is available.

Here is a reproduction https://stackblitz.com/edit/vitejs-vite-grnn2s?file=src%2Fworker.ts

[james-pre]

@robert-westenberger,

I cleaned up the RPC types in 0.16.0 and also changed the dependency range for @zenfs/core in @zenfs/dom 0.2.14 to be >= instead of ^, which means you can actually use core v0.13+ with it. Please let me know if upgrading both packages fixes the issue.

Thanks.

[robert-westenberger]

@james-pre Thank you for looking at this. I've updated core to 0.16.0 and dom to 0.2.14 and get the same issue. I've updated the reproduction:

https://stackblitz.com/edit/vitejs-vite-grnn2s?file=src%2Fworker.ts

[UnCor3]

@james-pre Yesterday, i started having this issue randomly and tried my best to fix the issue CI also passed can you check my pr #92 ?