Allow import of specific backends to reduce overall bundle size

[reklawnos]

Hi! Right now in src/core/backends.ts, all of the available backends are imported. This means that even if a user of BrowserFS does not use most of the backends, they still appear in the bundle.

I did a quick experiment to see what the impact would be of removing all of the backends from the bundle. The initial (unmodified) build had browserfs.min.js coming in a 247 kB, but with only the HTTPRequest backend imported, browserfs.min.js dropped to 112 kB.

The current recommendation on the homepage is to create a custom build with unneeded backends removed, but it seems like this could be made more ergonomic by moving the backends into separate modules/packages and letting the user of the library import the backends directly.

The API could be changed slightly, or just remain (mostly) the same to avoid breaking backward compatibility. For example, the BrowserFS.configure method could take a class for fs instead of a string:

import * as BrowserFS from 'browserfs';
import MountableFileSystemBackend from 'browserfs/MountableFileSystemBackend';
import InMemoryBackend from 'browserfs/InMemoryBackend';
import IndexedDBBackend from 'browserfs/IndexedDBBackend';

BrowserFS.configure({
    fs: MountableFileSystemBackend,
    options: {
      "/tmp": { fs: InMemoryBackend },
      "/home": { fs: IndexedDBBackend }
    }
  }

But to retain backward compatibility it could alternatively have a "registration"-based system:

import * as BrowserFS from 'browserfs';
import MountableFileSystemBackend from 'browserfs/MountableFileSystemBackend';
import InMemoryBackend from 'browserfs/InMemoryBackend';
import IndexedDBBackend from 'browserfs/IndexedDBBackend';

BrowserFS.registerBackend('MountableFileSystem', MountableFileSystemBackend);
BrowserFS.registerBackend('InMemoryBackend', InMemory);
BrowserFS.registerBackend('IndexedDBBackend', IndexedDB);

BrowserFS.configure({
    fs: 'MountableFileSystem',
    options: {
      "/tmp": { fs: 'InMemory' },
      "/home": { fs: 'IndexedDB' }
    }
  }

The current browserfs.js and browserfs.min.js bundles could use this under the hood, and a separate bundle could be made (named something like browserfs-core) with no backends included.

Additionally, if it makes sense, the repo could be restructured using Lerna. That way, the backends could be full-on separate packages that could be maintained, changed, and updated separately from the "core" while still living in the same repo. This would require types to be made available to those backend packages, which would incidentally fix #200.

Do any of these approaches make sense? I'd be happy to get to work on a PR that implements something like this!

[jvilk]

Glad to hear from you! I agree that BrowserFS could be made more modular. I have not done so yet because major consumers of the library have yet to desire the feature (I suspect because most are shipping BrowserFS alongside huge Emscriptened projects).

One challenge to modularity is that BrowserFS contains some helper classes that some file systems use. For example, SynchronousKeyValueStore is shared by both localStorage and in-memory backends. I suppose that these, too, could be moved into their own modules that are not part of the core, and consumers of BrowserFS that use a bundler will prevent wastefully duplicating the class.

Your suggestions make complete sense to me, and I would welcome a change like this as long as we do not break backwards compatibility with folks using the existing bundles. I just registered the browserfs NPM organization in preparation for our eventual modularity.

Additionally, if it makes sense, the repo could be restructured using Lerna.

Interesting; I had not heard of this project! I have resisted modularity in the past because I did not want to create and co-evolve a million separate Git repositories. It is already challenging with the separate process and path modules. Since large organizations (e.g., Babel) are using Lerna, I'd be happy to experiment with it for BrowserFS.

If it makes sense to you, I propose the following migration path to a modular BrowserFS:

1. Within the repository, your PR changes BrowserFS so that file systems only depend on a single browserfs-core module (and perhaps helper modules), and browserfs-core does not depend on any specific file systems. The build system will still create monolithic bundles, but all of the internal changes needed to support modularity will already be taken care of.
   • The import statements for these can still be ordinary relative paths, e.g. ../browserfs-core. The point is to prevent the need to 'reach into' specific submodules of browserfs-core; everything should be exposed from that one module.
2. A second PR will break up BrowserFS into separate modules (with or without Lerna; whichever makes more sense). The build script will need to be updated so that it continues building a monolithic browserfs.js and browserfs.min.js that maintain the existing API, and the tests will need to be updated to function appropriately with this new organization. A browserfs module should be one of these packages, which exports the current monolithic API for backwards-compatibility purposes.
   • Is this possible to do without publishing all of these modules to NPM? One of the challenges of modular development, for me, is co-evolving and testing the modules without publishing or manually npm linking individual packages.
   • Super optionally, it might be nice to merge in the bfs-path and bfs-process modules as @browserfs/path and @browserfs/process.
3. Once everything looks good, I will publish all of the modules to NPM.

I have to warn you that I am in a very busy period of my life. I am finishing up my PhD and preparing for a cross-country move in August. I will review any pull requests as soon as I can (please ping me when it is ready for review) and answer any questions you may have, but may not be actively coding alongside you that frequently. I have many plans for BrowserFS once I am settled in my new location!

With that said, I super super appreciate anything you are able to do that moves BrowserFS into a more modular state!

[reklawnos]

It's taken me a while to come back to this. Thank you for the super-detailed response!

That migration plan totally makes sense!

About this:

Is this possible to do without publishing all of these modules to NPM? One of the challenges of modular development, for me, is co-evolving and testing the modules without publishing or manually npm linking individual packages.

This is something that Lerna would make a lot easier. It has a lerna publish command that can bulk-publish all modules in a project. Another thing that is helpful is that Lerna (by default) keeps all of the packages fixed to the same version. Instead of updating individual package versions, they are all updated in lockstep when the repo is updated.

All that being said, it's also possible to avoid using Lerna entirely and just have users import the backends using paths (e.g. import IndexedDBBackend from 'browserfs/IndexedDBBackend';).

[jvilk]

All that being said, it's also possible to avoid using Lerna entirely and just have users import the backends using paths (e.g. import IndexedDBBackend from 'browserfs/IndexedDBBackend';).

The trouble with that approach is that we would have to have many files in the root of the browserfs package (which is also the root of this repository). So, if you are developing browserfs, your development folder will be filled with a bunch of files -- not good! (Unless there's some way around that?)

[ElvishJerricco]

FWIW, you can do this today just by instantiating file systems manually. This has the added benefit of being more type checked:

import FS from "browserfs/dist/node/core/FS";
import { DeviceFileSystem, Device } from "./."; // My custom FS
import { BFSCallback } from "browserfs/dist/node/core/file_system";
import MountableFileSystem from "browserfs/dist/node/backend/MountableFileSystem";

export function mkFS(devices: { [name: string]: Device }, cb: BFSCallback<FS>): void {
  DeviceFileSystem.Create({ devices: devices }, (e, dfs) => {
    if (e) {
      cb(e);
      return;
    }
    MountableFileSystem.Create({
      "/dev": dfs
    }, (e, _fs) => {
      if (e) {
        cb(e);
        return
      }

      const fs = new FS();
      fs.initialize(_fs);

      cb(undefined, fs);
    });
  });
}

Doing this allowed webpack to reduce the overhead of browserfs from over 300K to 80K

[james-pre]

Closing (stale). If you would like to reopen this issue, please do so by creating a new issue in the relevant repositories of @browser-fs