search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.12k stars 3.01k forks source link

change default hash function to blake2b-256 #6893

Open RubenKelevra opened 4 years ago

RubenKelevra commented 4 years ago

It would be nice if we could specify the default hash function in the config file. Currently, this looks to be hard-wired in the code to sha2-256.

I like to use blake2b-256 and CIDv1 all the time, because of more security and lower CPU cycles for hashing, allowing me to turn on hashing on reading.

Related to https://github.com/ipfs/go-ipfs/issues/6502

hsanjuan commented 4 years ago

As a personal opinion, I would be cautious with adding "it would be nice" things like this. It is an command option which can be set at will and as needed. Adding would also mean we should probably add configuration defaults for all other command options as long as someone finds it nice (essentially #6502). It introduces some ambiguity as the same command may result in different hashes with different daemons because the default is configured differently.

I see how this is useful, but personally, my approach to similar issues is to alias the right command with the right flags, which does not require a design discussion or any code in the target project. Integrations should probably explicitly set the options when not default. Allowing flips in what the default is, is not super polite, even if it likely won't break things in most places.

RubenKelevra commented 4 years ago

@hsanjuan alright, true.

Since we're currently using SHA2, how about switching the default to blake2b in the next release-cycle? It would reduce the time for processing new data significantly while adding more security.

In the SHA3 contest, BLAKE was considered more secure than Keccak, the SHA3 finalist.

hsanjuan commented 4 years ago

how about switching the default to blake2b in the next release-cycle?

I don't think this can happen on a minor release. It's a big behavior change. I am not sure about a potential 1.0.0.

RubenKelevra commented 4 years ago

how about switching the default to blake2b in the next release-cycle?

I don't think this can happen on a minor release. It's a big behavior change. I am not sure about a potential 1.0.0.

I was thinking about 0.5.0, since it breaks many stuff anyway.

It would increase the performance significantly. :)

https://github.com/ipfs/go-ipfs/issues/6776

Stebalien commented 4 years ago

0.5.0 isn't going to break a bunch of stuff. It will introduce a bunch of features, but chunking/hashing will stay the same for now.

When we finally switch to CIDv1 (almost ready), a new chunking algorithm (in progress), raw leaves (ready), inlining small blocks in CIDs (ready), etc. (maybe unixfs 2), then I expect we'll revisit the hash function (and likely switch to blake2b).

RubenKelevra commented 4 years ago

0.5.0 isn't going to break a bunch of stuff.

I'm sorry, I was imprecise...

I meant backwards compability will be broken, by introducing new features or removing others. Also wasn't meant to be critism, I really enjoying seeing new features and changes popping up on the horizon! :)

It will introduce a bunch of features, but chunking/hashing will stay the same for now. When we finally switch to CIDv1 (almost ready), a new chunking algorithm (in progress), raw leaves (ready), inlining small blocks in CIDs (ready), etc. (maybe unixfs 2), then I expect we'll revisit the hash function (and likely switch to blake2b).

TLS will replace SECIO, new chunking algorithm (if selected) will break compability with other clients adding the same files but getting different hashes. Etc.

That's what I meant. So sure, coalescing these changes makes total sense. I just saw a more than 2 times speedup for adding files just by switching to blake 2b without having any drawbacks. So I was just wondering why it's not on the timeline for 0.5.0.

Thanks for the explanation! :)

lzsaver commented 4 years ago

Or we may wait for blake3: https://github.com/multiformats/go-multihash/issues/121

RubenKelevra commented 3 years ago

Or we may wait for blake3: multiformats/go-multihash#121

While this progress doesn't sound bad - it's way too fresh to become any standard option. :)

Jorropo commented 1 year ago

blake2b caps out at ~1GiB/s on modern CPUs while sha256 with accelerators is ~2GiB/s, sha256 without accelerators can be much faster while blake2b is always not bad (given it's fast in software). I don't think this is a very good tradeoff, currently fastest IPFS implementation I have spend more CPU time on IO and only ~10% hashing data.