search

zboxfs / zbox

Zero-details, privacy-focused in-app file system.
https://zbox.io/fs/
Apache License 2.0
1.53k stars 74 forks source link

WebAssembly support #32

Closed skyne98 closed 4 years ago

skyne98 commented 5 years ago

Hey there! As far as I can see, ZboxFS has its ways of using the native FS and other services, therefore making me wonder if it can be successfully compiled to WASM and use as an in-memory secure file system. Are there any plans for it?

Thanks!

burmecia commented 5 years ago

Yes, WebAssembly support is definitely possible and it is in the roadmap. That sounds very interesting for me as well as we can run a virtual file system in browser! But to do that we need greatly reduce the binary size first, currently it is too large.

skyne98 commented 5 years ago

@burmecia, what is the binary size is not a problem for some projects? If you are choosing to write a web project with Rust now, you may be ready for big binary sizes. Is it possible to do it now, not considering the big binary sizes?

vladimiry commented 5 years ago

@burmecia WebAssembly can also be used at the backend/Node.js where the size of the dependency is less important than on the web.

burmecia commented 5 years ago

@skyne98 , @vladimiry After LLVM 8.0 was released, zbox WebAssembly support is finally possible. The hardest part was to integrate libsodium wasm with zbox. libsodium used emscripten to compile to wasm, while zbox use rust wasm-bindgen. Those 2 parts were hard to linked together without handful hacks. But now libsodium can be directly compiled to wasm by LLVM, which made wasm-bindgen easier to link to zbox.

The binary size now is around 2.3 MB (700KB+ gzipped) with some optimization, which is not too crazy. But I'd say still has some rooms to improve.

I am doing some polishing work now, hopefully will release it soon.

skyne98 commented 5 years ago

@burmecia, good news! I already played with libsodium and wasm myself, so I can confirm it works.

I think a small web demo using zbox for people to play with would be nice. Can spark interest in someone as well as explain basic concepts much better.

Also, have you tried passing a -Oz flag to LLVM directly?

burmecia commented 5 years ago

@skyne98 , yes I tried -Oz flag, but -Os flag seems produce slightly smaller binary size. So I made it default flag for release build.

A small web demo sounds a good idea. IndexedDB as the underlying storage, zbox runs in wasm, javascript as frontend provides POSIX-like async API, then we have a private filesystem running in browser! I will definitely put this on my task list.

skyne98 commented 5 years ago

Sounds like a plan :wink:

theduke commented 5 years ago

Is the the plan here to use localStorage as a backing storage layer for browser support? If yes that would be awesome

skyne98 commented 5 years ago

I think @burmecia mentioned using IndexedDB as backing storage. This will allow persisting the state between sessions (in theory).

theduke commented 5 years ago

The guarantees for IndexDB vs localStorage aren't really different, both are labeled as "temporary" and may be wiped out at any time. I do believe though that browsers are more forgiving with IndexDB both in terms of size restrictions and eviction.

There is a draft spec that allows requesting persistence that is not allowed to be wiped out arbitrarily. https://storage.spec.whatwg.org/#persistence

I believe this is implemented in FF and Chrome.

burmecia commented 5 years ago

The persistent web storage is fantastic but it is still experimental, some browsers even don't support it yet. For now, IndexedDB is used as local cache, it can persist between sessions but doesn't rely on it to provide durable storage.

burmecia commented 4 years ago

The wasm binding prototype is finally on https://zbox.io/try. Feel free to create a test repo and run it in browser. For now, only Chrome and Firefox is supported, the other browsers may have issues. Please raise issue on this repo: https://github.com/zboxfs/zbox-browser, as the wasm binding is now moved out to it.