New backend proposal: Filesystem FS

[billiegoose]

Correct me if I'm missing something obvious... but is there not a way to use BrowserFS with an actual file system? My reasoning being that BrowserFS has a number of advanced features (MountableFileSystem, InMemory, OverlayFS, ZipFS, and (with some polyfilling) XmlHttpRequest) that would be valuable in Node applications as well. It would also greatly simplify writing isomorphic JavaScript that runs in the browser or on the desktop.

[jvilk]

@wmhilton this has been proposed before. The issue is security. Without an authentication mechanism, it's a bit of a security issue. And when you get into authentication mechanisms, you're basically proposing that we implement some type of cloud storage.

You could hardcode the server to only listen on a local loopback, but that still gives anything running locally the same access to files as the user running the server process. And it seems confusing -- wouldn't people naturally "hack" it to listen on the web for remote access (or request that feature!), unknowingly opening up their computer to anyone on the net to modify files?

[billiegoose]

That is not what I'm suggesting at all. I wasn't even thinking of a server-client relationship. This would be a backend that is completely unusable in the browser.

Some backends are only usable in the browser, because they use features not available in Node and don't make a lot of since to polyfill. For instance, HTML5FS , WorkerFS, IndexedDBFS and LocalStorageFS. Maybe you could make them work in Node... but why?

Some could work in Node, if you polyfill certain features, and would be super useful. For instance, once global.fetch support lands in XmlHttpRequestFS it would probably work in node with the help of the isomorphic-fetch package. (Maybe it already works with the help of an XmlHttpRequest polyfill.)

Some backends seem like they should work equally well in browsers and in Node. InMemory, ZipFS and IsoFS for example.

The last category, is backends that only work in Node, that don't make any sense to implement on the browser. That's where FilesystemFS falls. It sounds crazy at first because the project description is literally "BrowserFS is an in-browser file system that emulates the Node JS file system API". But at some point BrowserFS became much more than that, and now code I've written for the browser is actually much more powerful and expressive than what I can get in Node.

I've got this awesome virtual file-system that works in the browser, and in service-workers, but in Node I have to use a different module (the native 'fs' one) to read and write to my hard drive. If BrowserFS also wrapped 'fs' itself, then I could write the same code for the browser and (say) a desktop CLI tool, and simply swap the config that gets passed to BrowserFS.

[jvilk]

@wmhilton Ahh, I see. Forgive me; I've had a few people suggest the server before, so I binned you in with them.

That does seem like a good idea. I would accept a pull request with that feature.

Checklist:

• [ ] Implement file system.
• [ ] Node FS wrapper does not get bundled in browser builds (browserfs.js / browserfs.min.js)
• [ ] Get the unit tests running and passing in Node on applicable file systems.

I can help with the last two items if someone else does the initial work on the first item.

[billiegoose]

Shucks, I was really hoping I was just missing something obvious. On the bright side, it should be about the easiest shim I've ever written.

How do I start? Make a new file in the backends folder, extend the BaseFileSystem class, add a constructor and a Create static method, and add it to core/backends.ts?

[jvilk]

Yeah, you've got it, but there's some subtleties here you should know.

You'll need to also make a File class that wraps the file descriptors that Node returns. Internally, BrowserFS uses File objects instead of file descriptors to simplify backend implementation. (Externally, users of BrowserFS see file descriptors/numbers.)

The Emscripten FS's file class is probably a good reference. Basically, the parts of the fs module that interact with file descriptors are moved into methods on a File object.

You'll also need to convert between Node's error objects and BrowserFS's error objects and Node's Stat objects and BrowserFS's Stat objects, which seems straightforward. Do a try/catch to catch and convert errors on synchronous methods, and wrap callbacks on async methods.

Finally, you'll need to make a test backend factory to test the file system.

Here's a checklist:

• [ ] New file in backends. It should use the same name as the class of the file system.
• [ ] Class for the file system with a static Create method and a constructor.
• [ ] Conversion code: Node's Error objects to BrowserFS error objects.
• [ ] Conversion code: Node's stats to BrowserFS stats.
• [ ] File class that wraps Node file descriptors.
• [ ] Wrap passed-in callbacks to convert error objects into BrowserFS error objects.
• [ ] Test factory in test/harness/factories that creates a Node FS backend instance.
  • If the Node FS gives the program full access to the entire file system, you'll need to wrap it in a FolderAdapter that points the tests at a useable scratch directory (e.g. path.join(os.tmpdir(), "bfs" + Math.random()).
• [ ] Passes unit tests once I get those running under Node.
• [ ] (For John): Configure bundler(s) to NOP out module somehow / replace with a stub so it isn't available in-browser / wasting bandwidth.

[nfriend]

I also would love a "real filesystem" backend! I was hoping this library would let my module target both a real filesystem (when the module is running as part of a Node application) and localStorage (when the module is running in the browser).

[TJKoury]

@wmhilton You ever start this? Going to take a hack at it if no one else is.

[billiegoose]

@TJKoury nope, I never started on it. I still think it would be super useful though!

[james-pre]

@wmhilton What is the status of this?

You could hardcode the server to only listen on a local loopback, but that still gives anything running locally the same access to files as the user running the server process.

Shouldn't #334 Fix this (since the uid/gid of the FS is configureable)?

[james-pre]

It should be possible to use MountableFileSystem.mount. If not, I can reopen the issue.