search

ipld / js-unixfs

UnixFS Directed Acyclic Graph for IPLD
Other
7 stars 4 forks source link

Automatically shard directories #43

Open alanshaw opened 1 year ago

alanshaw commented 1 year ago

https://github.com/ipld/js-unixfs/pull/41 added HAMT sharded directories support but it is opt in - you need to explicitly use the sharded directory writer.

In most IPFS implementations, the use of sharded directories (or not) depends on the byte size of the directory block with all the items in it. i.e. if the directory block exceeds a given size it automatically becomes a sharded directory.

rvagg commented 1 year ago

fwiw there's a test case that's shared across the other 3 main implementations that would be neat to have in here as well when you get to it - the boundary case for auto-sharding with defaults, where a particular CID is expected for the root of the test data before crossing the boundary and another CID is expected when you cross the boundary and get a sharded form. Alex recently got it working with js-ipfs-unixfs @ https://github.com/ipfs/js-ipfs-unixfs/pull/171 and there's some notes in there about the nuances of what's actually being measured for "size" (i.e. it's not actual block size, for historical reasons, but kind of an approximation).