SHA1_Final is deprecated in OpenSSL 3

[bkmgit]

When building with OpenSSL 3 get the following warnings:

/home/workflow/sogou-workflow/src/protocol/MySQLMessage.cc: In function ‘std::string protocol::__sha1_str(const std::string&)’:
/home/workflow/sogou-workflow/src/protocol/MySQLMessage.cc:256:18: warning: ‘int SHA1_Init(SHA_CTX*)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  256 |         SHA1_Init(&ctx);
      |         ~~~~~~~~~^~~~~~
In file included from /usr/include/openssl/x509.h:41,
                 from /usr/include/openssl/ssl.h:31,
                 from /home/workflow/sogou-workflow/src/protocol/MySQLMessage.cc:25:
/usr/include/openssl/sha.h:49:27: note: declared here
   49 | OSSL_DEPRECATEDIN_3_0 int SHA1_Init(SHA_CTX *c);
      |                           ^~~~~~~~~
/home/workflow/sogou-workflow/src/protocol/MySQLMessage.cc:257:20: warning: ‘int SHA1_Update(SHA_CTX*, const void*, size_t)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  257 |         SHA1_Update(&ctx, str.c_str(), str.size());
      |         ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/openssl/sha.h:50:27: note: declared here
   50 | OSSL_DEPRECATEDIN_3_0 int SHA1_Update(SHA_CTX *c, const void *data, size_t len);
      |                           ^~~~~~~~~~~
/home/workflow/sogou-workflow/src/protocol/MySQLMessage.cc:258:19: warning: ‘int SHA1_Final(unsigned char*, SHA_CTX*)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  258 |         SHA1_Final(sha1, &ctx);
      |         ~~~~~~~~~~^~~~~~~~~~~~
/usr/include/openssl/sha.h:51:27: note: declared here
   51 | OSSL_DEPRECATEDIN_3_0 int SHA1_Final(unsigned char *md, SHA_CTX *c);
      |                           ^~~~~~~~~~

Maybe should update to SHA512? Can make a pull request with this if of interest.

[Barenboim]

We know that. But 'SHA1()' function is slow.

[Barenboim]

MySQL authentification requires SHA1.

[Barenboim]

https://www.openssl.org/docs/man3.0/man3/SHA1.html

unsigned char *SHA1(const unsigned char *data, size_t count, unsigned char *md_buf);

This function is much slower in OpenSSL 3.0.

[bkmgit]

Is the EVP method also slow for SHA1?

[bkmgit]

A study on using SSE for Sha1 https://www.intel.com/content/www/us/en/developer/articles/technical/improving-the-performance-of-the-secure-hash-algorithm-1.html

A benchmark repo https://github.com/minaguib/sha1-benchmarks

However, this gets quite hardware specific. OpenSSL implementations for x86 seem to use vectorization, though ARMv8 could be improved to use NEON/SVE: https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-x86_64.pl https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-armv8.pl

[Barenboim]

OK, I will test EVP method too. I'v tested serveral SHA1 implementations including sha1 in redis. But the legacy functions of in OpenSSL are the fastest.

[bkmgit]

Thanks. Yes, native authentication uses double SHA1 https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_authentication_methods_native_password_authentication.html

Mysql has sha256 https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_authentication.html

Mariadb has alternative using ed25519 https://mariadb.org/history-of-mysql-mariadb-authentication-protocols/

[Barenboim]

EVP functions are also very slow... We will use the legacy functions so long as they are the fastest.

[Barenboim]

Thanks. Yes, native authentication uses double SHA1 https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_authentication_methods_native_password_authentication.html

Mysql has sha256 https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_authentication.html

Mariadb has alternative using ed25519 https://mariadb.org/history-of-mysql-mariadb-authentication-protocols/

We 'v not implemented MYSQL 8's caching sha2 authentication yet. Thank you for the documents.

[bkmgit]

That was very fast check. Will see if can update the previously linked benchmark, though if have suggestion for benchmark this would be helpful. Would like to compare:

• https://github.com/Mbed-TLS/mbedtls/blob/development/library/sha1.c
• https://developer.arm.com/documentation/ddi0596/2021-12/SIMD-FP-Instructions/SHA1C--SHA1-hash-update--choose--
• https://github.com/intel/ipp-crypto

[bkmgit]

@Barenboim Yes, it seems deprecated Sha1 implementation is faster from the test in https://github.com/bkmgit/sha1-benchmarks/tree/openssl-evp Nayuki's C implementation is quite fast, but speed will depend on compiler and architecture. Nayuki's assembly implementation can be updated to support newer architectures. Investigating why legacy implementation is faster.

[bkmgit]

SHA1 seems to be used in https://github.com/sogou/workflow/blob/master/src/protocol/MySQLMessage.cc#L256 and https://github.com/sogou/workflow/blob/master/src/manager/RouteManager.cc#L438 Profiled the benchmark and found that legacy version spends most time in a vectorized function however the updated version using EVP spends a lot of time in initialization. By moving EVP_DigestInit_ex(mdctx, EVP_sha1(), NULL); outside of the benchmarking loop, significant improvement is obtained

How soon do you expect to update to SHA2? Is there still interest in a dependency free implementation of SHA1 as explained at https://github.com/sogou/workflow/pull/967 ?

[bkmgit]

Original comparison:

Comparison with refactored benchmarking where EVP_DigestInit_ex(mdctx, EVP_sha1(), NULL) is outside of the benchmarking loop:

[Barenboim]

We are using this sha1 implementation in our nossl branch: https://github.com/sogou/workflow/blob/nossl/src/util/sha1.c https://github.com/sogou/workflow/blob/nossl/src/util/sha1.h This sha1 is widely used in open source projects including redis and libevent. You may also add it to you test.

[bkmgit]

Thanks for the pointer. An updated version of that implementation is available at https://github.com/noloader/SHA-Intrinsics though seems not to have been thoroughly vetted. Will try them both out.

OpenSSL EVP interface does seem like it will continue to have some performance overhead https://github.com/openssl/openssl/issues/19612 https://github.com/openssl/openssl/issues/17064

[bkmgit]

Nayuki and Git Sha1 implementations seem to have better performance: Code - results will depend on architetcure, compiler etc.

[bkmgit]

https://www.nist.gov/news-events/news/2022/12/nist-retires-sha-1-cryptographic-algorithm

[bkmgit]

Maybe also interesting is https://boringssl.googlesource.com/boringssl/

[Barenboim]

Maybe also interesting is https://boringssl.googlesource.com/boringssl/

We are fully compatible with boringssl.

[Barenboim]

Finally we are going to replace SHA1 with FNV-1a hash. Based on my test, the collision rate of FNV hash is even lower than SHA1 (mid 64bits), and has better performance as well. https://github.com/sogou/workflow/pull/1180

[bkmgit]

May also be of interest https://github.com/bcrypto/belt/blob/master/spec/0708Hash.tex

[Barenboim]

All the OpenSSL 3 SHA1 and RSA deprecated functions have be removed from the project. No we use FNV1a hash to generate hash key for each communication target, which in our scenario has a lower collision rate than SHA1 (64bits), and much better performance. Thank @bkmgit