blake2 tree hash usage and alternatives

xnox commented 2 days ago

Is your feature request related to a problem? Please describe

Currently OpenSSL blake2 implementation lacks tree hash options; and thus these are not available via openssl bindings in python. I am considering to remove support for the tree hash options from python, because they seem to be unused / not popular. However, global code search found https://github.com/aiidateam/aiida-core/blob/dd866ce816e986285f2c5794f431b6e3c68a369b/src/aiida/common/hashing.py#L102 which indeed does use blake2 python tree hash support

Describe the solution you'd like

If the tree hash options were no longer available in standard python, what would aiida-core do, and how painful would it be to switch to something else? Is cryptographic security required? Is it performance critical? Does it need to be in standard python?

Or is the need for any tree-hash that is just available in python and capable of creating hashes for otherwise unhashable objects?

Describe alternatives you've considered

Depending on true needs the following alternatives could be considered:

blake3 - it is merkle tree and parallel hash.... but i'm not sure how to use it as a drop in replacement for a tree hash
XXH3 from https://xxhash.com/ which is a magnitude improvement. Again not sure how to use it as a tree hash
TupleHash from SHA-3 family https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-185.pdf which I think could be a drop in replacement, but it seems like no bindings exist for it anywhere

Additional context

Note this is in part driven by the fact that on some python-fips implementations blake access in python is blocked; and thus likely making aiida-core not working.

unkcpz commented 1 day ago

The only issue I foresee was the old hash of nodes will changed if we use new hash algorithm. Which means the caching will not work for all nodes and it requires a database migration. We can try to keep this in mind and do the change when we ask for DB migration for something else.

xnox commented 1 day ago

I also have a secondary goal of keeping all of this working, even when python is in FIPS mode. As far as I can tell this usage really is "one-way compression function" rather than a "cryptographic hash". Thus I will experiment to still expose blake2 in python-fips, with "usedforsecurity=False" such that it can continue to be built. Then propose for aiida-core to pass that argument (when running on python3.9 or higher). Such that blake2 usage continues to be available, and aiida continious to work as is, and people can deploy it in FIPS environment (I have no idea if any do, but it seems easy enough to ensure they can).

aiidateam / aiida-core