Closed bkmgit closed 1 year ago
We know that. But 'SHA1()' function is slow.
MySQL authentification requires SHA1.
https://www.openssl.org/docs/man3.0/man3/SHA1.html
unsigned char *SHA1(const unsigned char *data, size_t count, unsigned char *md_buf);
This function is much slower in OpenSSL 3.0.
A study on using SSE for Sha1 https://www.intel.com/content/www/us/en/developer/articles/technical/improving-the-performance-of-the-secure-hash-algorithm-1.html
A benchmark repo https://github.com/minaguib/sha1-benchmarks
However, this gets quite hardware specific. OpenSSL implementations for x86 seem to use vectorization, though ARMv8 could be improved to use NEON/SVE: https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-x86_64.pl https://github.com/openssl/openssl/blob/master/crypto/sha/asm/sha1-armv8.pl
OK, I will test EVP method too. I'v tested serveral SHA1 implementations including sha1 in redis. But the legacy functions of in OpenSSL are the fastest.
Thanks. Yes, native authentication uses double SHA1 https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_authentication_methods_native_password_authentication.html
Mysql has sha256 https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_authentication.html
Mariadb has alternative using ed25519 https://mariadb.org/history-of-mysql-mariadb-authentication-protocols/
EVP functions are also very slow... We will use the legacy functions so long as they are the fastest.
Thanks. Yes, native authentication uses double SHA1 https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_authentication_methods_native_password_authentication.html
Mysql has sha256 https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_authentication.html
Mariadb has alternative using ed25519 https://mariadb.org/history-of-mysql-mariadb-authentication-protocols/
We 'v not implemented MYSQL 8's caching sha2 authentication yet. Thank you for the documents.
That was very fast check. Will see if can update the previously linked benchmark, though if have suggestion for benchmark this would be helpful. Would like to compare:
@Barenboim Yes, it seems deprecated Sha1 implementation is faster from the test in https://github.com/bkmgit/sha1-benchmarks/tree/openssl-evp Nayuki's C implementation is quite fast, but speed will depend on compiler and architecture. Nayuki's assembly implementation can be updated to support newer architectures. Investigating why legacy implementation is faster.
SHA1 seems to be used in https://github.com/sogou/workflow/blob/master/src/protocol/MySQLMessage.cc#L256 and https://github.com/sogou/workflow/blob/master/src/manager/RouteManager.cc#L438
Profiled the benchmark and found that legacy version spends most time in a vectorized function
however the updated version using EVP spends a lot of time in initialization.
By moving EVP_DigestInit_ex(mdctx, EVP_sha1(), NULL);
outside of the benchmarking loop, significant improvement is obtained
How soon do you expect to update to SHA2? Is there still interest in a dependency free implementation of SHA1 as explained at https://github.com/sogou/workflow/pull/967 ?
Original comparison:
Comparison with refactored benchmarking where EVP_DigestInit_ex(mdctx, EVP_sha1(), NULL)
is outside of the benchmarking loop:
We are using this sha1 implementation in our nossl branch: https://github.com/sogou/workflow/blob/nossl/src/util/sha1.c https://github.com/sogou/workflow/blob/nossl/src/util/sha1.h This sha1 is widely used in open source projects including redis and libevent. You may also add it to you test.
Thanks for the pointer. An updated version of that implementation is available at https://github.com/noloader/SHA-Intrinsics though seems not to have been thoroughly vetted. Will try them both out.
OpenSSL EVP interface does seem like it will continue to have some performance overhead https://github.com/openssl/openssl/issues/19612 https://github.com/openssl/openssl/issues/17064
Nayuki and Git Sha1 implementations seem to have better performance: Code - results will depend on architetcure, compiler etc.
Maybe also interesting is https://boringssl.googlesource.com/boringssl/
Maybe also interesting is https://boringssl.googlesource.com/boringssl/
We are fully compatible with boringssl.
Finally we are going to replace SHA1 with FNV-1a hash. Based on my test, the collision rate of FNV hash is even lower than SHA1 (mid 64bits), and has better performance as well. https://github.com/sogou/workflow/pull/1180
May also be of interest https://github.com/bcrypto/belt/blob/master/spec/0708Hash.tex
All the OpenSSL 3 SHA1 and RSA deprecated functions have be removed from the project. No we use FNV1a hash to generate hash key for each communication target, which in our scenario has a lower collision rate than SHA1 (64bits), and much better performance. Thank @bkmgit
When building with OpenSSL 3 get the following warnings:
Maybe should update to SHA512? Can make a pull request with this if of interest.