Investigate what open source Rust for WNFS libraries

[expede]

Related to #309

e.g. https://github.com/ramfox/spiral_ratchet

[matheus23]

https://github.com/ipld/libipld

@appcypher I promised you a more precise explanation of what we need. For WNFS v1 we'll need mostly dag-pb, for WNFS v2 we mostly need dag-cbor. In both cases we just need some kind of builder where at the end there's a bunch of bytes coming out of these libraries.

If they're not wasmifyable yet, let's figure out what it would take to make them wasmifyable/extract a wasmifyable core and possibly contribute back (but that'd be a step 2!).

[matheus23]

Perhaps it's useful to show you how we're using the equivalent libraries in TS land today:

WNFS v1 dag-pb usage

Encoding dag-pb nodes: https://github.com/fission-suite/webnative/blob/500bbb70e76105176174be4e528f33bf06e0b4c8/src/ipfs/basic.ts#L77-L86

Decoding dag-pb nodes: (somewhat hidden in here. js-ipfs actually does the heavy lifting of using dag-pb underneath here) https://github.com/fission-suite/webnative/blob/500bbb70e76105176174be4e528f33bf06e0b4c8/src/ipfs/basic.ts#L67-L75

WNFS v2 dag-cbor usage:

Encoding dag-cbor (some examples): https://github.com/fission-suite/webnative/blob/e1eed4e750c668a6f54d9be701dc17e286b9eff4/src/fs/data/public/publicNode.ts#L430-L442

Decoding dag-cbor (one example): https://github.com/fission-suite/webnative/blob/e1eed4e750c668a6f54d9be701dc17e286b9eff4/src/fs/data/public/publicNode.ts#L445-L453

---

These libraries may use sockets, files and more somewhere else, but I'm hoping these functions for en/decoding stuff we can at least extract out into pure functions.

[appcypher]

Thanks @matheus23. There are useful contexts.

[appcypher]

As for things that concern WNFS, the skip ratchet implementation is the only public one, I know of so far, that comes from the community. I will keep adding to this list as I discover more of them. There are also other open source Rust libraries from Protocol Labs and other orgs/individuals that are not necessarily WNFS specific. Most are IPFS/IPLD related.

Community WNFS Projects

• Skip Ratchet - https://github.com/ramfox/spiral_ratchet

  I have not worked with this yet since I'm still focusing on pub WNFS. I need to investigate this properly when I get started on private WNFS. It is still actively maintained and I'm in talks with the author on what works and what needs to be implemented.

Non WNFS Specific Projects

• IPLD DAG - https://github.com/ipld/libipld

  This is an official IPLD implementation and it is actively maintained. This is vital to a pub WNFS implementation since we use it to construct the public filesystem.

• MultiHash - https://github.com/multiformats/rust-multihash

  Also an official implementation of multihash. I've not had to use it yet but I'm thinking it will be handy for CID constructing in tests.

• HAMT

  • https://github.com/raviqqe/hamt-rs
  • https://github.com/michaelwoerister/hamt-rs

  There are multiple options here with varying levels of maintenance. We need this for private WNFS implementation.

I will keep updating this as I learn more about the ecosystem.

[expede]

Closing!