Implement FBL based splitting of large zarr chunks

d70-t / ipldstore

zarr backend store using IPLD datastructures

MIT License

6 stars 7 forks source link

Implement FBL based splitting of large zarr chunks #8

Open d70-t opened 2 years ago

d70-t commented 2 years ago

The IPFS peer to peer layer has a hard limit of 2MB block size. In order to support larger data chunks, we need to split them up. The IPLD-native way to do so seems to be the FBL ADL.

IPLDStore should be modified to split up large bytes-Objects into FBL structures.