search

genonullfree / teleporter

Teleporter is a simple application for sending files from Point A to Point B
BSD 3-Clause "New" or "Revised" License
30 stars 6 forks source link

Dramatic increase in time for hashing (delta overwrite) on MacOS #61

Closed genonullfree closed 2 years ago

genonullfree commented 2 years ago

Perform some binary profiling to determine exact case of slowdown.

Affected versions: v0.6.0, v0.5.5, suspect the following culprits:

Ex Linux machine orig:

 =>  666.433M of  666.433M (100.00%) done! Time: 37.06s Speed: 144.094 Mbps

Linux machine delta overwrite:

 done! Time: 1.18s Speed: 3530.437 Mbps

Ex MacOS machine (M1Max) orig:

 =>    1.145G of    1.145G (100.00%) done! Time: 19.52s Speed: 493.502 Mbps

MacOS machine (M1Max) delta overwrite:

 done! Time: 84.77s Speed: 111.626 Mbps
genonullfree commented 2 years ago

At a glance, it seems that Blake3 is highly optimized for intel x86, seems sus

genonullfree commented 2 years ago

Swapping out Blake3 for AHash and FxHash yields the following differences:

Aarch64: Blake3 == Time: 84.91s Speed: 111.626 Mbps AHash == Time: 7.82s Speed: 1339.507 Mbps (~12x faster) FxHash == Time: 24.37s Speed: 390.689 Mbps xxHash == Time: 9.61s Speed: 1041.839 Mbps

x86_64: Blake3 == Time: 1.95s Speed: 5331.465 Mbps AHash == Time: 9.28s Speed: 592.385 Mbps (~9x slower) FxHash == Time: 24.85s Speed: 222.144 Mbps xxHash == Time: 10.52s Speed: 533.146 Mbps

As sad as those stats are for x86_64, I think it does bring a bit more parity to other architecture speeds....

Maybe it could be optional?

genonullfree commented 2 years ago

Ok, doing a new comment for the --release versions (thanks @wcampbell0x2a !)

Aarch64: Blake3 == Time: 3.14s Speed: 3125.516 Mbps (current default) AHash == Time: 252.15ms Speed: inf Mbps FxHash == Time: 575.60ms Speed: inf Mbps XXH64 == Time: 254.68ms Speed: inf Mbps XXH3 == Time: 480.89ms Speed: inf Mbps

x86_64: Blake3 == Time: 1.08s Speed: 5331.465 Mbps AHash == Time: 342.29ms Speed: inf Mbps FxHash == Time: 511.00ms Speed: inf Mbps XXH64 == Time: 372.59ms Speed: inf Mbps XXH3 == Time: 318.64ms Speed: inf Mbps

I like how the code gets cleaned up for AHash/FxHash/xxHash, so I'll probably end up merging in AHash, but that'll cause another minor version bump, oops.

Cyan4973 commented 2 years ago

If collision ratios of hash functions matters for your application, you might want to have a look at this comparison exercise .

genonullfree commented 2 years ago

@Cyan4973 Yes it does! I would certainly like my file transfer utility to not have to really worry about hash collisions when possible. Thank you for commenting, your link is a great resource that I missed earlier.