search

jvilk / BrowserFS

BrowserFS is an in-browser filesystem that emulates the Node JS filesystem API and supports storing and retrieving files from various backends.
Other
3.07k stars 220 forks source link

Web Workers #89

Closed SpiritedDusty closed 9 years ago

SpiritedDusty commented 10 years ago

Is there a way I could use BrowserFS's localStorage backend while being in a web worker? Since web workers don't have access to localStorage, I don't expect BrowserFS to work in a web worker with the localStorage backend.

jvilk commented 10 years ago

Hello again. At the moment, you cannot for the reasons you specified. Additionally, BrowserFS does not function in a WebWorker (there are a few hardcoded references to window), but that is on the roadmap -- along with other goodies (e.g. caching and syncing in-memory file system changes to asynchronous storage).

Are you currently trying to use BFS in a WebWorker? From within a WebWorker, there are synchronous interfaces to WebSQL (Safari), HTML5 FIleSystem (Chrome/Blink Opera), and soon IndexedDB (Firefox only; other browsers only support the async API). I hope to support them all eventually.

jvilk commented 10 years ago

In addition, in the future, it should be trivial to support asynchronous access to localStorage and other backends hosted in the main browser thread, as I can leverage WebWorker messaging for that.

SpiritedDusty commented 10 years ago

I'm trying to use BFS inside a WebWorker, but I'm trying to get cross browser support, which is why I wanted to use localStorage. BFS seems like the only library out there that has a filesystem that can hook into Emscripten's FS.

jvilk commented 10 years ago

Since localStorage is big enough for you... I assume you're trying to store <5MB of data, correct?

Sounds like a good use case for a filesystem that keeps its contents in-memory, but asynchronously flushes changes to backing storage (e.g. localStorage). I think a lot of Emscripten applications would benefit from a file system that does this from within a WebWorker.

Do you have something that currently builds with Emscripten and runs inside a WebWorker?

SpiritedDusty commented 10 years ago

Yes I'm not storing any more than 5MB of data and yes my project builds with Emscripten and runs inside of a WebWorker. I had to use a WebWorker for my project because occasionally the Emscripten program will stall and that would freeze up the DOM.

jvilk commented 10 years ago

I had to use a WebWorker for my project because occasionally the Emscripten program will stall and that would freeze up the DOM.

Yup, I know. Using a WebWorker gives you much better performance and responsiveness, especially if it's a game interacting with the oddly computationally expensive 2D canvas.

Is this open source and easily buildable?

SpiritedDusty commented 10 years ago

As in the Emscripten part? All there is for the Emscripten part is just Lua. I'm making an emulator for a game and part of it relies on Lua.

jvilk commented 10 years ago

kk. Just wondering. When I go to tackle this, it's nice to have a sample application that I can embed BFS into for testing, since I haven't tangoed with Emscripten in a WW yet.

I can probably gin something up on my own, though, when it comes time for it.

jvilk commented 10 years ago

Side note: BrowserFS currently works in a WebWorker (I just refactored out any direct references to window), although there are no synchronous web storage backends for that environment yet.

With that said, the XHR backend works fine, and I'm sure the ZipFS backend will work, too. I'll eventually modify our unit tests to run the tests in a WW.

EDIT: Current dev version only; not the stable version. I'm not ready to make the current dev version 'stable' yet, but it should still work fine.

SpiritedDusty commented 10 years ago

Have you tested this in a WebWorker with Emscripten yet?

jvilk commented 10 years ago

Yes, that's the only place that I have tested it so far. Note that I did not test it exhaustively yet.

I have only tested synchronous operations in the XmlHttpRequest file system, and no others.

It's also very possible that async operations in a WW are broken; I added a WebWorker setImmediate polyfill, but I did not test it yet.

jvilk commented 10 years ago

Whoops; I forgot to commit some of the changes.

The current revision on master has the remaining WebWorker fixes.

SpiritedDusty commented 10 years ago

I guess I'll wait for the stable version. I currently have a hacked together a cheap way of packaging a folder into JSON and loading it in using Emscripten's FS API. It works but just kind of messy. BrowserFS would make things a lot more clean.

jvilk commented 9 years ago

I just added WorkerFS to BrowserFS, which lets you mount file systems from the main thread in a web worker (and visa-versa). This will enable you to write files to localStorage from a Web Worker, but since WebWorker communications are asynchronous, you cannot integrate this into the Emscripten file system.

I will open an issue for a "AsyncSync" file system wrapper, which will load the entire contents of an asynchronous file system into a synchronous file system at bootup, and will sync writes asynchronously to the backing storage as they occur. This will let Emscripten mount any asynchronous storage.

jvilk commented 9 years ago

WorkerFS isn't in the latest BFS release, btw. But it will be in the next one, which should occur today or tomorrow.