Implement BLAKE2b checksums

[lukateras]

SHA256 is rather slow at checksumming large files (tangentially related to #619), would be nice to have something that is both fast and future-proof.

Use case would be to nix-hash large (> 4GB) files to check if already present in Nix store. This takes a while with SHA256, and would probably take at least 2x less time with BLAKE2b.

[lukateras]

I would like to send a patch if this is approved (no new dependencies).

[edolstra]

Well, we already have SHA-512, which is much faster than SHA-256 (at least in the current implementation).

[lukateras]

BLAKE2b is about 1.5x faster than SHA-512 (similar to margin between SHA-256 and SHA-512), while also being way simpler (compare https://github.com/openssl/openssl/blob/master/crypto/sha/sha512.c at 694 SLOC with https://github.com/openssl/openssl/blob/master/crypto/blake2/blake2b.c at 241 SLOC). See https://blake2.net/.

It was also introduced in coreutils, see b2sum(1) and http://lists.gnu.org/archive/html/coreutils/2015-05/msg00048.html.

(For anyone who wants to study the function, here's cleaned up source code: https://github.com/yegortimoshenko/sylpheed/blob/d7a91f82e5785992e3716529b63d8530560a5479/src/blake2b.c)

[xzfc]

Another promising candidate is BLAKE3 (which appeared recently in Jan 2020). It is 6x faster than sha512 on my machine on some random 91M file from /nix/store.

Benchmark

```sh $ du -sh /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi 91M /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi $ hyperfine 'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' Benchmark #1: b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi Time (mean ± σ): 45.8 ms ± 4.4 ms [User: 142.5 ms, System: 10.4 ms] Range (min … max): 41.4 ms … 58.9 ms 62 runs Benchmark #2: sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi Time (mean ± σ): 274.6 ms ± 19.2 ms [User: 258.6 ms, System: 15.8 ms] Range (min … max): 254.9 ms … 309.2 ms 11 runs Summary 'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' ran 5.99 ± 0.72 times faster than 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' $ b3sum --version b3sum 0.1.1 $ sha512sum --version sha512sum (GNU coreutils) 8.31 Packaged by https://NixOS.org Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Ulrich Drepper, Scott Miller, and David Madore. $ grep 'model name' /proc/cpuinfo | head -n1 model name : Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz ```

[tnias]

Probably duplicate of #1133.

Gentoo uses blake2b since end of 2017. (source: wiki page on manifest files and mailing list annoucement)

Archlinux has blake2 support. (source: arch wiki PKGBUILD and b2sums in PKGBUILD(5))

[stale[bot]]

I marked this as stale due to inactivity. → More info

[figsoda]

blake3 seems like a good candidate

[stale[bot]]

I marked this as stale due to inactivity. → More info