Improve integration between Parquet and IPFS

datonic / datadex

📦 Serverless and local-first Open Data Platform

http://datadex.datonic.io

MIT License

222 stars 14 forks source link

Improve integration between Parquet and IPFS #6

Closed davidgasquez closed 2 years ago

davidgasquez commented 2 years ago

Partition parquet files in smaller chunks to fit with IPLD
Use a trickle DAG: ipfs add --cid-version 1 --raw-leaves -r --trickle

davidgasquez commented 2 years ago

This won't scale, since it puts more weight on users. It might be interesting to explore it in a different setting, a decentralized data warehouse on top of IPFS.