SHA1 hash is wrong over S390x architecture on Debian

[eribertomota]

The Debian Project has a CI system to test any package over the following architectures: amd64, armel, armhf, i386, ppc64el and s390x.

The following test fails on s390x:

dcfldd if=test.txt hash=md5,sha1,sha256,sha384,sha512 2>&1 | grep -C 20 bc0e4b74695142e0a0bdae87aea310d7078866cb

I attached the file test.txt to this issue. The grep shown above will test the SHA1 result. The other hashes are right. See below the result of the test over amd64 WITHOUT the grep:

Total (md5): 92994b0ce292a217e3e3bc31b639e565

Total (sha1): bc0e4b74695142e0a0bdae87aea310d7078866cb

Total (sha256): 02fd428a4671925e4ca61541b9fac648f4ccdccad65602bfd3256ba14e59489c

Total (sha384): 53b8374607a8258de4173265bfbfa6120093fd42090a92fd589cf2c6c16b4e421b5135777714976713f7949715720a83

Total (sha512): 969a39bf47b5f12d81121084f19cb5ae250e0c0ea5b7c6d82cf08131acde8e1955d137612c2f6b255e25b0e28f96f93586f90f06965cb8f719ed7fbdd95cc8d4

Now, over s390x:

Total (md5): 92994b0ce292a217e3e3bc31b639e565

Total (sha1): 68afccc873eb954866097c08d4ad9d9643735f2c

Total (sha256): 02fd428a4671925e4ca61541b9fac648f4ccdccad65602bfd3256ba14e59489c

Total (sha384): 53b8374607a8258de4173265bfbfa6120093fd42090a92fd589cf2c6c16b4e421b5135777714976713f7949715720a83

Total (sha512): 969a39bf47b5f12d81121084f19cb5ae250e0c0ea5b7c6d82cf08131acde8e1955d137612c2f6b255e25b0e28f96f93586f90f06965cb8f719ed7fbdd95cc8d4

Only SHA1 differ between both architectures. Executing the sha1sum command over the file on s390x will produce the right hash:

(sid_s390x-dchroot)eriberto@zelenka:~$ sha1sum dcfldd-1.7.1/debian/tests/test.txt 
bc0e4b74695142e0a0bdae87aea310d7078866cb  dcfldd-1.7.1/debian/tests/test.txt

test.txt.gz

[davidpolverari]

Hi @eribertomota!

The Debian Project has a CI system to test any package over the following architectures: amd64, armel, armhf, i386, ppc64el and s390x.

All those architectures but s390x are little-endian (well, some of them are bi-endian, though little-endian by default, but I digress), thus my first guess is that is an endianess-related problem.

Only SHA1 differ between both architectures. Executing the sha1sum command over the file on s390x will produce the right hash:

By skimming through the code, I saw that the SHA1 implementation relies on some #defines (RUNTIME_ENDIAN and WORDS_BIGENDIAN) to determine processor endianess during runtime and during the build, while the other hashing implementations don't use them, as can be verified by the following grep on the project root dir:

find . -type f -exec egrep -H '(RUNTIME_ENDIAN|WORDS_BIGENDIAN)' {} \;

I found that WORDS_BIGENDIAN should be defined by configure.ac by using the AC_C_BIGENDIAN macro [^1]. I couldn't find anything about the other (RUNTIME_ENDIAN) macro, but I think we should do something like OpenLDAP did to enable determining endianess during runtime ^2.

I'll have a deeper look at it, by trying to generate the same output as s390x on amd64 by "hardcoding" the defines at strategic places for debugging. Anyway, if I eventually come to a conclusion, I won't be able to test it on s390x. But then we can make a tentative branch for you to test it there when we have something, if it is ok to you.

[^1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.62/autoconf.pdf, p.75.

[eribertomota]

Hi @davidpolverari

I won't be able to test it on s390x. But then we can make a tentative branch for you to test it there when we have something, if it is ok to you.

Yes! Thanks a lot! Go ahead!

[davidpolverari]

I'll have a deeper look at it, by trying to generate the same output as s390x on amd64 by "hardcoding" the defines at strategic places for debugging. Anyway, if I eventually come to a conclusion, I won't be able to test it on s390x. But then we can make a tentative branch for you to test it there when we have something, if it is ok to you.

Indeed, by making a test on amd64 with the changes below, I got dcfldd to do the opposite it should do when hashing SHA1, treating it as a big-endian arch. I think that by using the AC_C_BIGENDIAN macro on configure.ac, we will be able to solve the problem on s390x. Below you can find the diff for the changes I used to test and the test results after using them:

patch

--- dcfldd-1.7.1.orig/src/sha1.c
+++ dcfldd-1.7.1/src/sha1.c
@@ -51,6 +51,8 @@

 #include <string.h>

+#define WORDS_BIGENDIAN
+
 #include "sha1.h"

 #ifndef lint

Results (by doing the opposite from the native architecture)

Total (md5): 92994b0ce292a217e3e3bc31b639e565

Total (sha1): 68afccc873eb954866097c08d4ad9d9643735f2c

Total (sha256): 02fd428a4671925e4ca61541b9fac648f4ccdccad65602bfd3256ba14e59489c

Total (sha384): 53b8374607a8258de4173265bfbfa6120093fd42090a92fd589cf2c6c16b4e421b5135777714976713f7949715720a83

Total (sha512): 969a39bf47b5f12d81121084f19cb5ae250e0c0ea5b7c6d82cf08131acde8e1955d137612c2f6b255e25b0e28f96f93586f90f06965cb8f719ed7fbdd95cc8d4

I will prepare the new branch and upload it soon.

[davidpolverari]

I have prepared a tentative fix (https://github.com/resurrecting-open-source-projects/dcfldd/commit/cb54a40d831999fff4158e3ac1c2c561c3dea59f) on the bugfix/issue-11 branch. Please, let me know if it worked on s390x after you try it. We can later merge it into master if it does. Thanks!

Note: maybe adding only build-time endianness configuration will not be enough, as several posts over the Internet report inconsistencies about involved compiler flags. We also might need to add runtime endianness detection support (by using the same approach as OpenLDAP). I will work on that too.

[davidpolverari]

I added another commit to the bugfix/issue-11 branch that adds a configure option (--enable-runtime-endian-check) to enable runtime endianness checking, in addition to the build-time check. It is disabled by default.

[eribertomota]

Worked fine on s390x!

[davidpolverari]

Great! Thanks for the test! I made the runtime endianness checking the default during the build. Should the user want to disable it, it can be done by using the --disable-runtime-endian-check during ./configure. I will merge it on master now.