search

multiformats / cid

Self-describing content-addressed identifiers for distributed systems
Other
414 stars 78 forks source link

CID length and identity hashes #21

Open Stebalien opened 6 years ago

Stebalien commented 6 years ago

Moved from: https://github.com/ipfs/go-ipfs/issues/4918 as this isn't go-ipfs specific and will affect the spec.

Basically, we'd like to allow inlining small blocks into CIDs (using the identity hash function) for performance reasons. However, the larger the block we allow to be inlined, the less user friendly CIDs get. Unfortunately, we have to pick a "default inlined size" up front or we'll end up changing a bunch of hashes later.

Open questions:

@whyrusleeping @kevina @diasdavid @vmx @kyledrake

Stebalien commented 6 years ago

@kevina's options:

  1. Don't set a hard limit on CID digest size, but by default id hashes will have a maxium digest length (and thus content length) of 64 bytes
  2. Set a hard limit of 128 bytes on digest length (to keep things from getting to out of hand, but also to not artificially limit our options) but limit id hashes to 64 bytes by default.
  3. Set a hard limit of 64 bytes on digest length and thus limit id hashes to this length

@Stebalien's additional options:

  1. Soft limit of 38 bytes for the entire CID. That'll allow a base32 encoded CID to fit in a domain name segment.
  2. Soft limit of 42 bytes for the entire CID. That's what we use for inlining peer IDs (although we may want to reduce this to 38 given the DNS restriction.

Unfortunately, my options limit the utility of this feature. However, they do increase the usability.

kevina commented 6 years ago

It is important to note that the 64 bytes comes from the maximum size modern crypro. hashes output (512 bits). If we set the limit lower than this will we prevent the option of using the all the output bits.

Stebalien commented 6 years ago

We definitely can't set a lower hard limit, but we could set a lower auto-inline limit. That really depends on how likely we feel we are to move to a larger hash sometime soon.

vmx commented 6 years ago

I don't have a strong opinion on the limits. Though I'm not really sure about this whole data inlining. It will make the whole stack more complicated. Currently it's always "CID + data" and then if will become "CID + maybe data, depending on the CID". This is a huge change a lot of components need to learn about.

whyrusleeping commented 6 years ago

@vmx i'm not entirely sure what you mean. Conceptually, its pretty simple. We're just allowing the 'hash' function of the CID to be f(x) = x. Everything else works exactly the same. The thing this enables though, is a cool optimization where we don't have to actually store data for CIDs using this particular 'hash' function.

vmx commented 6 years ago

@whyrusleeping I'm thinking in in code. Currently it's:

  1. get request via CID
  2. ask storage for this CID
  3. return the thing the storage returned

Then a new step between 1 and 2 is introduced:

I'm not sure how bad this really is. If you all think that's not really a big deal, that's fine for me :)

Stebalien commented 6 years ago

So, our plan is to just modify the block service to "do the right thing". That is, when you try to put a block with a CID that uses the identity hash, it'll just throw it away. When you try to get the block, it'll extract it from the CID.

Currently we have to create indirect, large CIDs even for really tiny objects, files, and directories.

richardschneider commented 6 years ago

It was easy to change my block service to support getting an inlined CID.

However, I have some concerns when putting.

kevina commented 6 years ago

this should be optional/experimental behaviour. Otherwise, pre-existing tests fail because the CID is different

Why?

richardschneider commented 6 years ago

@kevina putting a small block in a test, blockService.Put(byte[] { 0x01 }), generates a different CID if CID inlining is enabled. By definition CID v1 must be used, whereas without CID inlining, CID v0 can be used.

kevina commented 6 years ago

@richardschneider the automatic use of identity hashes will require a command line flags that is not enabled by default. See https://github.com/ipfs/go-ipfs/pull/4910 for a proof-of-concept implementation.

richardschneider commented 6 years ago

@kevina Thanks, did not know about --id-hash-limit option. So question 1 is answered.

Does the limit specify the number bytes in (1) the data block size or (2) the identity hash digest size or (3) the CID binary length?

Also, is id an alias for the identity hash algorithm?

kevina commented 6 years ago

Does the limit specify the number bytes in (1) the data block size or (2) the identity hash digest size or (3) the CID binary length?

(2) the identity hash digest size

Also, is id an alias for the identity hash algorithm?

Yes

richardschneider commented 6 years ago

Thanks @kevina. You have provided enough info for me to implement the putting. Cheers!

richardschneider commented 6 years ago

What should be the behavior when the block service remove is called with an inline Cid?

I'm thinking this a no-op and no error is returned.

Stebalien commented 6 years ago

I'm thinking this a no-op and no error is returned.

I agree. Personally, I'd prefer it if moved to being idempotent over deletes for both performance and usability reasons.

richardschneider commented 6 years ago

@kevina Would you mind commenting on https://github.com/richardschneider/net-ipfs-engine/issues/20