[2021 Theme Proposal] Increase max block size / defualt to blake2b-256

lizelive commented 4 years ago

Note, this is part of the 2021 IPFS project planning process - feel free to add other potential 2021 themes for the IPFS project by opening a new issue or discuss this proposed theme in the comments, especially other example workstreams that could fit under this theme for 2021. Please also review others’ proposed themes and leave feedback here!

Theme description

One thing that limits IPFS usage for large datasets is its very small block size. 256kb is too small, and the largest size (at least for files) is 1mb. For better disk performance a minimum of 4mb would be needed. This will also drastically improve performance when using cloud backings for data storage.

Hypothesis

Large datasets for repeatable research are a pain to move around, ipfs can fix this.

Vision statement

allow bitswap to handel 4mb files.

Why focus this year

The useage is growing and moving to a 4mb default will result in a 16x reduction in overhead. Also makes it performant to use hdd as backing.

Example workstreams

Please list relevant workstreams, development milestones, and a high-level timeline for these efforts.

Other content

Please include links to other relevant content, notes, etc.

welcome[bot] commented 4 years ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

"Priority" labels will show how urgent this is for the team.
"Status" labels will show if this is ready to be worked on, blocked, or in progress.
"Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

atopal commented 4 years ago

Hi @lizelive

Thank you for this. It looks like your submission might be a possible solution for issue #69 (Scalability) and not a theme in itself. I'm going to close this issue, but can you please comment on that issue with your input?

ipfs / roadmap