Mounting filesystems

[mstade]

It seems there's only an in-memory filesystem included with wasm-git (totally makes sense) but if I'm not completely mistaken it also looks like this can quite easily be replaced simply by mounting a filesystem at a given path? Digging through the FS object I see this:

Is there any documentation you can point to on how to implement this interface, and is this all that's needed to plug in a different kind of filesystem?

What I'd like to do is actually write to disk, not memory, when my application is running on a local machine (in Electron, hence #2, but I can do this in Node just as well) but when it's running in a browser I obviously would use an in-memory filesystem. I don't want to just dump one big blob, I'd actually like to write proper files that could be used with a regular git client outside of my app as well.

[petersalomonsen]

If you build it yourself you should be able to include support for the file-systems that come with emscripten:

https://emscripten.org/docs/api_reference/Filesystem-API.html#file-systems

In your case you probably want NODEFS

I haven't tested this, but will try as soon as I get time. Thanks for feedback :-)

[mstade]

Brilliant, I'll see if I can create a build for this, thanks for the tips!

[samdenty]

Doesn't work with NodeFS, breaks creating .git/config.lock files.

Works better with emscripten NODERAWFS=1, but still has some issues.

https://github.com/emscripten-core/emscripten/pull/10162 fixes the first issue

[petersalomonsen]

Does it work with emscripten-core/emscripten#10162 ?

I did some testing myself and found the reason for the lock failing to be that after renaming a file, the file still exist with the old name:

https://github.com/emscripten-core/emscripten/issues/10628

[samdenty]

@petersalomonsen Lock file renaming doesn't fail with NODERAWFS. But it fails with readdir 'Invalid Arguments', which #10162 fixes.

Then failed in libgit2/src/unix/map.c#58, with failed to mmap. Could not write data: No such device

[petersalomonsen]

So this should fix the locking issue for NODEFS:

https://github.com/emscripten-core/emscripten/pull/10631

Now I get the same mmap error as you get. Will look into this soon, unless someone is quicker than me :-)

[petersalomonsen]

this fixes the mmap error:

https://github.com/emscripten-core/emscripten/pull/10669

after that you'll get permission errors on library_fs.js open method and library_nodefs.js setattr. A dirty workaround to this is to make sure that the mknod call in the open method always have write permission ( node = FS.mknod(path, mode | 0o200, 0); ), and before the truncateSync in setattr also give write permission. I was able to clone in NODEFS this way, but for a PR to emscripten I'll need something better than changing permissions.

[petersalomonsen]

I've successfully cloned using NODEFS with the changes committed here: https://github.com/petersalomonsen/wasm-git/commit/f9ed551f37d2f06408960f91c7edd2e72df4fb90

[nachoab]

Would it be possible to use the new WasmFS and mount the OPFS fs?

I'm including this flags in the build command, but it's creating a lg.js file that doesn't contain the opfs implementation

-s WASMFS -s FORCE_FILESYSTEM

[petersalomonsen]

Thanks, I wasn't aware of this, but it would be interesting to check if it works out of the box. Otherwise I would assume there are minor adjustments needed to support it. I'll have a look in the near future.

[petersalomonsen]

Now compiling with the latest emscripten here: https://github.com/petersalomonsen/wasm-git/pull/58

Had to increase the stack size to make it work.

Not sure if it actually use WASMFS. Do I have to add the -s WASMFS switch?

[nachoab]

Now compiling with the latest emscripten here: #58

Had to increase the stack size to make it work.

Not sure if it actually use WASMFS. Do I have to add the -s WASMFS switch?

Including -s WASMFS is supposed to include the WASMFS JS libraries, but I'm not seeing any of the methods of the opfs js library being included in the exported js unless explicitly added on EXPORTED_RUNTIME_METHODS. But even then, is not useful since FS doesn't include a way to load the OPFS backend, so I guess FS it's just running on memory (with different API so maybe that's making git calls fail)

I asked how to test this here and was suggested to take a look at the tests. But those are tests for the C implementation and don't show how to mount a FS

[petersalomonsen]

Maybe it is too early? According to the tracking page here, many of the essential parts are still in status TODO: https://github.com/orgs/emscripten-core/projects/1/views/1?filterQuery=

[nachoab]

Yeah your right, I'll give a try in the future

[MeydanOzeri]

any update on this topic ? im also trying to create a custom file system

[petersalomonsen]

Yeah I see that there is progress here: https://github.com/orgs/emscripten-core/projects/1/views/1?filterQuery=

So maybe it's worth a try to integrate this with wasm-git in the near future.

[zavx0z]

Any progress?

[petersalomonsen]

Just tried to compile with WASMFS enabled, but seems like it doesn't work quite yet. From the progress tracking pages in the emscripten project ( see previous message ), there are several needed features still missing. So I don't expect this to work yet.

[matbee-eth]

Just tried to compile with WASMFS enabled, but seems like it doesn't work quite yet. From the progress tracking pages in the emscripten project ( see previous message ), there are several needed features still missing. So I don't expect this to work yet.

Which browser did you test with btw?

[petersalomonsen]

Just tried to compile with WASMFS enabled, but seems like it doesn't work quite yet. From the progress tracking pages in the emscripten project ( see previous message ), there are several needed features still missing. So I don't expect this to work yet.

Which browser did you test with btw?

I just enabled the WASMFS flag, and tried running the tests ( using chrome browser ). Might be that I have not configured it correctly, but I've started some work on this now and also a Wasi build.. You can follow my progress in this PR here: https://github.com/petersalomonsen/wasm-git/pull/88

Currently been focusing on the Wasi build which is also a quite interesting use case.

[matbee-eth]

Just tried to compile with WASMFS enabled, but seems like it doesn't work quite yet. From the progress tracking pages in the emscripten project ( see previous message ), there are several needed features still missing. So I don't expect this to work yet.

Which browser did you test with btw?

I just enabled the WASMFS flag, and tried running the tests ( using chrome browser ). Might be that I have not configured it correctly, but I've started some work on this now and also a Wasi build.. You can follow my progress in this PR here: #88

Currently been focusing on the Wasi build which is also a quite interesting use case.

Do you think you could scaffold out some WASMFS tests that I could run against? Understanding you're currently focusing on WASI, I'm not terribly familiar with wasm-git codebase, but I would like to see how far I can get with my minor requirement of filesystem features.

[petersalomonsen]

Just tried to compile with WASMFS enabled, but seems like it doesn't work quite yet. From the progress tracking pages in the emscripten project ( see previous message ), there are several needed features still missing. So I don't expect this to work yet.

Which browser did you test with btw?

I just enabled the WASMFS flag, and tried running the tests ( using chrome browser ). Might be that I have not configured it correctly, but I've started some work on this now and also a Wasi build.. You can follow my progress in this PR here: #88 Currently been focusing on the Wasi build which is also a quite interesting use case.

Do you think you could scaffold out some WASMFS tests that I could run against? Understanding you're currently focusing on WASI, I'm not terribly familiar with wasm-git codebase, but I would like to see how far I can get with my minor requirement of filesystem features.

Here's a simple WASMFS test that passes: https://github.com/petersalomonsen/wasm-git/pull/88/commits/844fa31cb2f03b419c54cc7fcac73e499bd78e65

So it seems to handle simple operations as commit and log, so looks promising.

[petersalomonsen]

So I run into problems when mounting WasmFS backends. This is the error I see: https://github.com/emscripten-core/emscripten/issues/18112 which is currently a blocker that needs to be fixed.

[fredericrous]

let's all add thumbs up 👍 to issue https://github.com/emscripten-core/emscripten/issues/18112 then, to try to make it visible

[petersalomonsen]

Some updates on what I think are the next steps here: https://github.com/petersalomonsen/wasm-git/pull/88#issuecomment-2097417075

[petersalomonsen]

So before I start with a new attempt on WASMFS/OPFS, I would like to change to ES6 modules and I also need to replace the Karma test runner ( which is not maintained anymore ).

I have done this in the following PR: https://github.com/petersalomonsen/wasm-git/pull/97

Note that changing to ES6 might break client implementations, but still I think it's better to also encourage these to start using ES6. I would like to not support CommonJS moving forward, but let me know if there are any strong opinions on this.

[fredericrous]

I've been using vite and vitest for a year now, on several projects. I'm super happy with these. especially with the fact that configuration is minimal and the tools are fast.

No need for commonjs support. git wasm is bleeding edge technology, I expect it to only support es6

[ngocdaothanh]

I hope that wasm-git will be able to use ZenFS: https://github.com/zen-fs/core