search

maandree / sha3sum

[Feature complete] SHA-3 and Keccak checksum utility
https://codeberg.org/maandree/sha3sum
ISC License
174 stars 51 forks source link

sha3sum gives different hash between file on same partition or on a mount point #32

Closed Bonnietwin closed 2 years ago

Bonnietwin commented 2 years ago

I found an effect when downloading sqlite and tryong to calculate the sha3-256sum hash but I have shown it applies to any file.

If I have a file on the same partition as I am working on I get one sha3-256 sum and that matches the result from the website that the file is being downloaded from. If that same file is copied to my nfs fileserver which is on a mount point on /mnt then the sha3-256 hash is different. This applies consistently irrespective of the file being used and where on the mountpoint filesystem the file is stored.

If I calculate the sha256sum on the same files then the result is the same on the partition being worked on and the mountpoint.

Something is changing in the file that the sha3-256sum algorithm includes but that the sha256sum algorithm doesn't.

I have downloaded sqlite several times in the past and did not have this problem. The files I download always go to the fileserver and the hash is then checked on the file there. In the past the sha3-256 hash always matched the one from the website.

I am running arch linux.

Let me know if any additional information is needed.

maandree commented 2 years ago

Are you using the latest version of libkeccak, does rolling back to the previous version of libkeccak change anything? How big is the file, and how big is the I/O block size?

Bonnietwin commented 2 years ago

Arch Linux is a rolling distro so is very up to date usually. sha3sum is 1.2.1 libkeccak is 1.3

I rolled back to 1.2.2 of libkeccak and the problem is no longer there. The sha3sum is correct for the file wherever it is located.

The sqlite tar file is 2.9MiB and the bacula tar file is 4.0MiB I will also do a test with a much smaller file and see if there is a difference and find out what the I/O block sizes are for the files evaluated. Rolling back to the previous version of libkeccak is a working solution for me until the issue is solved.

Bonnietwin commented 2 years ago

I am not sure if I have found what you were asking for with the I/O Block Size but after googling, I have run the stat command on the file. On the same partition as I was working on that gave the correct sha3 hash the IO Block number was 4096 On the nfs mounted fileserver (running on a raid array) the number was 262144

Bonnietwin commented 2 years ago

I forgot to mention in the last post that the partition I was working on that gave the correct sha3 hash is also a raid array, so both locations are raid array based.

maandree commented 2 years ago

Great, then I'm rather sure where the problem is located. I'll get back to you when I've looked at it.

Bonnietwin commented 2 years ago

I installed the 1.3 version of libkeccak again and ran the test on different file sizes.

The same different sha3sum occurred with files of 1016.5KiB and 158.5KiB but with a file of 2.7KiB the sha3sum was the same for the file in both locations.

maandree commented 2 years ago

I'm certain it is fixed now, would you mind checking with the lasted commit to libkeccak?

Bonnietwin commented 2 years ago

Sorry for delay but I was a bit busy today.

Just patched the source and installed it using the Arch Linux Build System and tested it out on the sqlite package and now I get the same sha3-256sum as they have on the sqlite website.

Thanks very much. Marking this issue as closed. Look forward to it ending up in the Arch Linux system but the patched version works so no problems any more for me.