search

ipfs-rust / ipfs-embed

A small embeddable ipfs implementation
344 stars 47 forks source link

Default block size too small for go-ipfs interop #72

Open rklaehn opened 3 years ago

rklaehn commented 3 years ago

The maximum block size in go-ipfs is 4 mb, so it is possible to create a block in go-ipfs that can not be bitswapped.

Either we should increase the default block size to 4mb (exact size is somewhere in the maze of repos that is go-ipfs), or at least document that the default size will not work seamlessly for go-ipfs interop.

Note that at Actyx we might have some old blocks from the go-ipfs times laying around which are >1mb.

dvc94ch commented 3 years ago

I'm completely fine with increasing it, it can just never be decreased again once it is increased. also I'm still unhappy about protocol labs thinking it's ok to make this "implementation defined". @vmx can protocol labs finally decide on an arbitrary number without falling into analysis paralysis?

vmx commented 3 years ago

I'm on it trying to find it out once for good.

Stebalien commented 3 years ago

The maximum block size bitswap supports is actually 2MiB. However, if you're using the default UnixFS encoder, there's some framing overhead so the maximum chunk size is less than 2MiB.

The following rules should cover it:

  1. Don't chunk larger than 1MiB. If a block is slightly larger than 1MiB, that should be fine.
  2. Support blocks up to 2MiB.
Stebalien commented 3 years ago

I stand corrected. The max message size is actually 4MiB. I still recommend not creating blocks larger than 1MiB+epsilon, but it would be safer to support blocks up to 4MiB.

Stebalien commented 3 years ago

(note: supporting blocks up to 2MiB should still be enough given the recommended 1MiB chunk size max)

dvc94ch commented 3 years ago

It's not quite that easy. js-ipfs uses a 16k block size [0]. I'd appreciate this to be added to some spec somewhere.

Stebalien commented 3 years ago

Not quite. That's the target message size, not the max message size. Basically, if the message size is less than 16KiB, we'll try to pack more into the message. Once it hits 16KiB, we'll send it off.

But yeah, we do need to spec this (both in IPLD as a "recommendation" and in bitswap).

dvc94ch commented 3 years ago

@Stebalien what's the status of this? Is there a spec or PR I can reference in the code?