search

hpc / mpifileutils

File utilities designed for scalability and performance.
https://hpc.github.io/mpifileutils
BSD 3-Clause "New" or "Revised" License
167 stars 64 forks source link

switch ddup hashing algorithm to xxHash #538

Open theAeon opened 1 year ago

theAeon commented 1 year ago

improves performance massively and moots #535

Signed-off-by: Andrew Robbins andrew@robbinsa.me

theAeon commented 1 year ago

Just a quick update on the viability of this:

it seems that xxhash runs so quickly that the threads start clobbering each others' output in the outfile. might need a bit of parallelism reworking that wasn't necessary w/ a secure hash like sha256.