search

filecoin-station / zinnia

Runtime for worker nodes executing modules in a resource-limited sandbox
https://crates.io/crates/zinnia
Other
19 stars 2 forks source link

Access content of individual files in the CAR stream fetched via Lassie #246

Open bajtos opened 1 year ago

bajtos commented 1 year ago

This is a follow-up for https://github.com/filecoin-station/zinnia/pull/243, see also https://github.com/filecoin-station/roadmap/issues/19

Lassie supports only one response format: CAR file. We should enable modules to work with the data stored inside this CAR file.

For example, if the CAR file contains UnixFS with a single file (e.g. https://bafybeib36krhffuh3cupjml4re2wfxldredkir5wti3dttulyemre7xkni.ipfs.dweb.link), then a module should have a way how to obtain the content of that file:

My most famous drawing, and one of the first I did for the site

There are different ways how to approach this.

  1. We can leverage IPFS packages for the browser to load the CAR file into an in-memory block store, interpret it as UnixFS and export file content.

  2. Loading large CAR files into an in-memory block store is inefficient when the content is large. Zinnia should provide a built-in block store backed by the filesystem (persisted in a file stored in the CACHE DIR), including APIs to load CAR streams into this block store in a streaming way. Related: https://github.com/filecoin-station/zinnia/issues/192

  3. Since Zinnia does not support 3rd party modules yet (see #209), I think we should either work on that story to allow modules to leverage existing JS packages for handling UnixFS content or else add a built-in version of UnixFS. This built-in version can be a vendored bundle of an existing JS package or a think JS API backed by Rust UnixFS implementation.

    Unfortunately, it seems that all Rust UnixFS crates have been abandoned by now. 😢

Based on the above, I think we should start with the first option - find a way to leverage existing JS browser-compatible tooling for both UnixFS and CAR loader/store. Then we can replace the JS version of CAR loader/store with a built-in Zinnia block store & CAR loader. Finally we can look into improving DX of using JS UnixFS.

juliangruber commented 1 year ago

For the first step, would that be to vendor the packages into zinnia, write a tutorial on how to vendor them into a module, or implement 3rd party dependency import?

bajtos commented 1 year ago

For the first step, would that be to vendor the packages into zinnia, write a tutorial on how to vendor them into a module, or implement 3rd party dependency import?

I prefer to write a tutorial or proper documentation on how to vendor 3rd-party packages into a module.

There is a limit on how many packages we can bundle into zinnia and still keep well-maintained.

I had a good experience with deno bundle (we are using it for zinnia:assert), but that command has been deprecated by Deno. We can look into other options out there, e.g. esbuild, swc or even rollup.