BrowserFS as a Browserify builtin

[wjordan]

If you're interested in integrating BrowserFS with the Browserify ecosystem, one extremely interesting feature that could result from such integration is that the BrowserFS package could not only emulate the Node.js filesystem API approximately, but the package could be used as a Browserify builtin for the 'fs' package directly (see e.g. how level-fs-browser works for abstracting synchronous fs operations to a levelDB store). In this way, existing code and libraries using the 'fs' package could function equally well in the browser without any code changes using BrowserFS (and whatever further configuration is needed to configure the filesystem backend).

[jvilk]

One issue with this approach is that we have our own Buffer and path modules that are API-compatible with Node's, so we would have to tell Browserify to not use its builtins. This is a hard requirement for the Buffer module, as backends can supply their own Buffer backends to reduce copying overhead (see the BufferCore classes).

If we can do that, then the next challenge is figuring out the configuration story, and mapping something like e.g. a JSON configuration into a configured fs module.

[wjordan]

First step might be to take a deep look at feross/buffer and substack/path-browserify, the packages currently used as default built-ins for Browserify? At first glance it seems that there might be some significant overlap between the features provided by these node API-compatible packages and the corresponding implementations within this project (and to the extent there are differences, it might be possible to integrate the differences into these other projects, if they represent new features or extra use-cases).

[jvilk]

I think I can do this with a browserfs-fs package that marks BrowserFS as a peerDependency, and merely exports BrowserFS.BFSRequire('fs').

EDIT: Same w/ path and the gang. You wouldn't be able to use Browserify's Buffer or process though.

[jvilk]

There are particular reasons why I can't support browserify's default modules:

• Their Buffer is an array with functions dangling off of it. Our Buffer does not support indexing operations because I wished to have decent data representations on older browsers without TypedArray support.
  • We actually use an internal representation of an ArrayBuffer, CanvasPixelArray, or an array of 32-bit integers depending on the environment. Browserify's module will either use an ArrayBuffer or an array of bytes -- the latter of which has decent performance but terrible memory usage on older browsers (4X or 8X memory overhead, depending on how smart the browser is).
• We support process.chdir(), they do not. This has implications for path, which must process . into the current directory.

I could potentially monkey-patch their path and process, but I wouldn't be able to use their Buffer.

[wjordan]

hmm, interesting. Thanks for looking into this. From your summary, I'm still not sure whether the differences in path, process and Buffer are fundamental design/API differences in the respective modules, or whether these differences could be eventually integrated into the Browserify ecosystem with some future work, either via:

1. upstream pull requests to the current default browserify builtins, e.g.:
   • Buffer (feross/buffer): optionally use CanvasPixelArray depending on the browser environment
   • Buffer: support higher-performance set()/get() functions through an API extension;
   • Buffer: fallback internal implementation of ArrayBuffer for older browsers;
   • process (defunctzombie/node-process): support process.chdir;
   • path: (not sure what's needed here?)
2. make the path/process/Buffer modules available as alternate builtins themselves, in addition to fs.

Now that #107 is done, a first step towards making fs available as a reusable builtin could be to split out the BrowserFS versions of path, process and Buffer into distinct npm packages, that could then be swapped in as replacements for the standard browserify builtins if they offer additional functionality that can't be merged upstream. For example, if the Buffer module you've developed is in fact more browser-compatible and/or more performant than feross/buffer, then this could make it possible for projects beyond BrowserFS to take advantage of the gains in this alternate implementation as well.

After this step is done, you could have the fs module depend on its sibling Node-API modules by simply requiring that an API-compatible set of builtins be available within Browserify's environment, which would eliminate the need to maintain the library-swapping logic in the current all-in-one package, and allow the Node-API implementations to evolve and be reused independently from one another.

[jvilk]

I'm warming up to this idea. I think this is doable. Some thoughts:

Buffer

Some work would need to be done to decouple our Buffer implementation from the file systems. Right now, some file systems reach in and grab the underlying buffer to avoid copying. These can be changed, of course, to recognize some well-known buffer types and casting as needed. I'd likely make util functions for this sort of thing.

Regarding performance, feross/Buffer likely has better runtime performance in newer browsers, as it likely has less abstraction overhead. It also supports the index [] operator, whereas my Buffer cannot (without resorting to something ludicrous, like defining numerical properties for each index of the Buffer). I'm sure many people would prefer to use that over my Buffer if they couldn't care less about legacy browsers (where legacy = doesn't support ArrayBuffer).

feross/Buffer can't use CanvasPixelArray, btw, as it doesn't support the index [] operator. It's primarily useful for IE9 support.

One problem: Since my file systems use Buffer itself, something would need to tell it what Buffer to use. I don't know if Browserify has a concept of redirectable peerDependencies (e.g. "when these packages require("Buffer"), use this package"). I'd prefer not to resort to something hacky, like making a fs.tellMeWhatBufferToUse(), as that would make file system code really messy. Any suggestions would be appreciated.

EDIT: If I decouple the Buffer, I could also potentially get NodeJS support...

EDIT2: I just realized there's no need to require('buffer'), as Browserify can take care of acting as if it's a global. So maybe this isn't a problem?

Path

No work needed. I could potentially deprecate my path module. Their path properly calls process.cwd(), so it has chdir support.

Process

I support chdir, stdout, stderr, and stdin. I actually re-implemented Node's event emitters, although I should really just reuse Node's versions. These are not used by BrowserFS itself, though; just some applications that use it (OK, maybe just DoppioJVM). Thus, I should be able to use the community's process with no issue.

[jvilk]

This also brings me back to #46, which might be a good idea.

[jvilk]

@wjordan you can now do this, BUT you need to subscribe to the entire BrowserFS ecosystem for now for purely engineering-related reasons.

Here's an example repository:

https://github.com/jvilk/bfs-browserify-test

I am planning to separate out the modules, letting you grab only 'what you need', but I want to wait until TypeScript 1.5 is released. Right now, there are terribly annoying issues with Node TypeScript typings with TypeScript 1.5 that I've manually hacked around in the main BrowserFS repository, and I don't wish to do so for each module's individual repository. I'm using TypeScript 1.5 because tsify requires me to, and using tsify makes my build scripts / life so much simpler.

Once those issues are resolved, I will happily undertake the small amount of refactoring required to decouple the modules from BrowserFS. And until that day comes... the solution in the linked repository is your best bet for using BrowserFS with browserify.

Let me know if you have any q's.

[wjordan]

The test looks great! This setup looks perfectly usable within Browserify environments, at this point any further module refactoring will probably just bring improvements to the final bundle size.

I actually did a little bit of experimenting with separating out the modules over the weekend- see browser-buffer, browser-path, browser-process, and BrowserFS#browserfs-node-modules.

For creating an npm package that can be loaded into another browserify build, since I was running into the issues with Browserify struggling on re-parsing the already-bundled package, I actually had better luck not bundling the module's source files, and instead just transpiling the TypeScript using tsc and making all of the *.js sources available directly from the npm package. I'm not sure if the source maps made it through to the final build though.

Using commit bb8769 on bfs-browserify-test which references these isolated modules for all builtins except fs, the minified test build comes to around 205k.

[jvilk]

@wjordan this is now done. Please look at bfs-browserify-test, which now has a build configuration that uses BrowserFS's fs module with Browserify's implementation of process, path, and Buffer. Also, take note of the new information in the README (and note that I stopped using bfs-fs).