wnfs-wg / rs-wnfs

Rust implementation of the WebNative FileSystem (WNFS) specification
https://github.com/wnfs-wg
Apache License 2.0
147 stars 24 forks source link

Get all the cids that were created/removed as a result of an operation that changes the HAMT #87

Closed ehsan6sha closed 1 year ago

ehsan6sha commented 2 years ago

NB: Feature requests will only be considered if they solve a pain

Summary: Besides returning just the new root cid after each operation, also return all the cids that are created or removed as a result

Problem

Right now, when we do an operation in WNFS, it returns the new root cid as a result. To understand which cids are added or removed we need to compare the whole HAMT with the last version and traverse manually.

Impact

This gives the flexibility to the protocol developers on top of WNFS to choose how they want to keep the replicates of WNFS synced together.

Solution

Besides returning just the new root cid, it return added or removed cids.

Detail

Is your feature request related to a problem? Please describe. Let's say a protocol is developed on top of WNFS that keeps the WNFS-generated cids backed up on multiple untrusted devices (like how Filecoin is chunking and keeping the chunks). Right now, there is a way to keep the whole HAMT synced easily, but there is no easy way to back up part of the HAMT and always sync on some untrusted devices.

Describe the solution you'd like When performing an operation that changes the root cid of WNFS HAMT, also get a list of cids that were added and a list of cids that were removed as a result of that specific action. For example, cids that we added as a result of mkdir and those cids that are no longer belonging to the new version and are deprecated.

Describe alternatives you've considered Traversing the HAMT from the root by the protocol on top of WNFS itself and finding the differences between old and new versions of HAMT.

Additional context Discussed initially on IPFS discord in #wnfs channel with Boris

matheus23 commented 1 year ago

Ehsan and I have discussed this live previously and we agreed that solving this on the BlockStore level is probably the best way to do this. In rs-wnfs, we're abstracting over the BlockStore details, and our MemoryBlockStore is only meant for testing, so that'd be a feature to look for in iroh, beetle or kubo.