search

randombit / botan

Cryptography Toolkit
https://botan.randombit.net
BSD 2-Clause "Simplified" License
2.54k stars 562 forks source link

'Target specific option mismatch' with amalgamation #1386

Closed fgsalomon closed 6 years ago

fgsalomon commented 6 years ago

Hello, until now I've been using Botan 2.0.1 with gcc 5.4 with --single-amalgamation-file without specifying -march options and haven't had any problem. Now I want to use version 2.3.0 but I'm getting target specific option mismatch errors. Which compilation options should I use? I tried with -m64 -maes -mpclmul -mssse3 -mrdrnd -msse2 -mrdseed but building still fails.

Compilation error samples:

In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/immintrin.h:71:0,
                 from /usr/lib/gcc/x86_64-linux-gnu/5/include/x86intrin.h:46,
                 from /usr/include/x86_64-linux-gnu/c++/5/bits/opt_random.h:33,
                 from /usr/include/c++/5/random:50,
                 from /usr/include/c++/5/bits/stl_algo.h:66,
                 from /usr/include/c++/5/algorithm:62,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.h:11,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:8:
/usr/lib/gcc/x86_64-linux-gnu/5/include/shaintrin.h:87:1: error: inlining failed in call to always_inline ‘__m128i _mm_sha256rnds2_epu32(__m128i, __m128i, __m128i)’: target specific option mismatch
 _mm_sha256rnds2_epu32 (__m128i __A, __m128i __B, __m128i __C)
 ^
/home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:51141:58: error: called from here
       STATE0 = _mm_sha256rnds2_epu32(STATE0, STATE1, MSG);
                                                          ^
In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/x86intrin.h:37:0,
                 from /usr/include/x86_64-linux-gnu/c++/5/bits/opt_random.h:33,
                 from /usr/include/c++/5/random:50,
                 from /usr/include/c++/5/bits/stl_algo.h:66,
                 from /usr/include/c++/5/algorithm:62,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.h:11,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:8:
/usr/lib/gcc/x86_64-linux-gnu/5/include/tmmintrin.h:185:1: error: inlining failed in call to always_inline ‘__m128i _mm_alignr_epi8(__m128i, __m128i, int)’: target specific option mismatch
 _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N)
 ^
/home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:51154:44: error: called from here
    STATE1 = _mm_alignr_epi8(STATE1, TMP, 8); // ABEF
                                            ^
In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/x86intrin.h:41:0,
                 from /usr/include/x86_64-linux-gnu/c++/5/bits/opt_random.h:33,
                 from /usr/include/c++/5/random:50,
                 from /usr/include/c++/5/bits/stl_algo.h:66,
                 from /usr/include/c++/5/algorithm:62,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.h:11,
                 from /home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:8:
/usr/lib/gcc/x86_64-linux-gnu/5/include/smmintrin.h:166:1: error: inlining failed in call to always_inline ‘__m128i _mm_blend_epi16(__m128i, __m128i, int)’: target specific option mismatch
 _mm_blend_epi16 (__m128i __X, __m128i __Y, const int __M)
 ^
/home/fgsalomon/myproject/deps/Botan/botan_source_desktop/botan_all.cpp:51153:47: error: called from here
    STATE0 = _mm_blend_epi16(TMP, STATE1, 0xF0); // DCBA

Amalgamation output:

INFO: ./configure.py invoked with options "--amalgamation --single-amalgamation-file --disable-modules=pkcs11"
   INFO: Platform: OS="Linux" machine="x86_64" proc="x86_64"
   INFO: Guessing target OS is linux (use --os to set)
   INFO: Guessing to use compiler gcc (use --cc to set)
   INFO: Detected CPU model "Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz" in /proc/cpuinfo
   INFO: Guessing target processor is a x86_64/x86_64 (use --cpu to set)
   INFO: Canonicalized CPU target x86_64 to x86_64/x86_64
   INFO: Auto-detected compiler version 5.4
   INFO: Target is gcc:5.4-linux-x86_64-x86_64
   INFO: Skipping (dependency failure): certstor_sqlite3 sessions_sqlite3
   INFO: Skipping (disabled by user): pkcs11
   INFO: Skipping (incompatible CPU): aes_armv8 pmull sha1_armv8 sha2_32_armv8
   INFO: Skipping (incompatible OS): darwin_secrandom getentropy win32_stats
   INFO: Skipping (no enabled compression schemes): compression
   INFO: Skipping (not requested): dyn_load
   INFO: Skipping (requires external dependency): bearssl boost bzip2 lzma openssl sqlite3 tpm zlib
   INFO: Loading modules: adler32 aead aes aes_ni aes_ssse3 aont aria asn1 auto_rng base base64 bcrypt bigint blake2 block blowfish camellia cascade cast cbc cbc_mac ccm cecpq1 certstor_sql cfb chacha chacha20poly1305 chacha_rng chacha_sse2 clmul cmac codec_filt comb4p cpuid crc24 crc32 cryptobox ctr curve25519 des dev_random dh dl_algo dl_group dlies dsa eax ec_gfp ec_group ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 gcm gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea idea_sse2 iso9796 kasumi kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion locking_allocator mac mce mceies md4 md5 mdx_hash mgf1 misty1 mode_pad modes mp newhope noekeon noekeon_simd numbertheory ocb ofb par_hash passhash9 pbes2 pbkdf pbkdf1 pbkdf2 pem pgp_s2k pk_pad poly1305 poly_dbl prf_tls prf_x942 proc_walk pubkey rc4 rdrand rdrand_rng rdseed rfc3394 rfc6979 rmd160 rng rsa salsa20 seed serpent serpent_simd sessions_sql sha1 sha1_sse2 sha1_x86 sha2_32 sha2_32_x86 sha2_64 sha3 shacal2 shacal2_simd shacal2_x86 shake shake_cipher simd siphash siv skein sm2 sm3 sm4 sp800_108 sp800_56a sp800_56c srp6 stateful_rng stream streebog system_rng threefish threefish_avx2 tiger tls tls_cbc tss twofish utils whirlpool x509 x919_mac xmss xtea xts
   INFO: Assuming CPU is little endian
   INFO: Assuming unaligned memory access works
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Writing amalgamation header to botan_all.h
   INFO: Writing amalgamation header to botan_all_internal.h
   INFO: Writing amalgamation source to botan_all.cpp
   INFO: Botan 2.3.0 (revision git:9474deacd67433908dec38a409892a334aab679d) (release dated 20171002) build setup is complete

Thanks.

webmaster128 commented 6 years ago

Same thing happens on current master when Botan's Makefile is used:

./configure.py --amalgamation --single-amalgamation-file --disable-modules=pkcs11 && make
   INFO: ./configure.py invoked with options "--amalgamation --single-amalgamation-file --disable-modules=pkcs11"
   INFO: Guessing target OS is linux (use --os to set)
   INFO: Guessing to use compiler gcc (use --cc to set)
   INFO: Detected CPU model "Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz" in /proc/cpuinfo
   INFO: Guessing target processor is a x86_64/x86_64 (use --cpu to set)
   INFO: Found sphinx-build (use --without-sphinx to disable)
   INFO: Found rst2man (use --without-rst2man to disable)
   INFO: Canonicalized CPU target x86_64 to x86_64/x86_64
   INFO: Auto-detected compiler version 7.2
   INFO: Target is gcc:7.2-linux-x86_64-x86_64
   INFO: Skipping (dependency failure): certstor_sqlite3 sessions_sqlite3
   INFO: Skipping (disabled by user): pkcs11
   INFO: Skipping (incompatible CPU): aes_armv8 pmull sha1_armv8 sha2_32_armv8
   INFO: Skipping (incompatible OS): darwin_secrandom getentropy win32_stats
   INFO: Skipping (no enabled compression schemes): compression
   INFO: Skipping (not requested): dyn_load
   INFO: Skipping (requires external dependency): bearssl boost bzip2 lzma openssl sqlite3 tpm zlib
   INFO: Loading modules: adler32 aead aes aes_ni aes_ssse3 aont aria asn1 auto_rng base base64 bcrypt bigint blake2 block blowfish camellia cascade cast cbc cbc_mac ccm cecpq1 certstor_sql cfb chacha chacha20poly1305 chacha_rng chacha_sse2 clmul clmul_ssse3 cmac codec_filt comb4p cpuid crc24 crc32 cryptobox ctr curve25519 des dev_random dh dl_algo dl_group dlies dsa eax ec_gfp ec_group ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 gcm gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea idea_sse2 iso9796 kasumi kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion locking_allocator mac mce mceies md4 md5 mdx_hash mgf1 misty1 mode_pad modes mp newhope nist_keywrap noekeon noekeon_simd numbertheory ocb ofb par_hash passhash9 pbes2 pbkdf pbkdf1 pbkdf2 pem pgp_s2k pk_pad poly1305 poly_dbl prf_tls prf_x942 proc_walk psk_db pubkey rc4 rdrand rdrand_rng rdseed rfc3394 rfc6979 rmd160 rng rsa salsa20 seed serpent serpent_simd sessions_sql sha1 sha1_sse2 sha1_x86 sha2_32 sha2_32_x86 sha2_64 sha3 shacal2 shacal2_simd shacal2_x86 shake shake_cipher simd siphash siv skein sm2 sm3 sm4 socket sp800_108 sp800_56a sp800_56c srp6 stateful_rng stream streebog system_rng threefish threefish_avx2 tiger tls tls_cbc tss twofish utils whirlpool x509 x919_mac xmss xtea xts
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Writing amalgamation header to botan_all.h
   INFO: Writing amalgamation header to botan_all_internal.h
   INFO: Writing amalgamation source to botan_all.cpp
   INFO: Botan 2.4.0 (revision git:8c1340482d9c069de93b762da27588f785af51f8) (unreleased undated) build setup is complete
g++ -fstack-protector -m64 -pthread  -fPIC -fvisibility=hidden -std=c++11 -D_REENTRANT -O3 -momit-leaf-frame-pointer -Wall -Wextra -Wpedantic -Wstrict-aliasing -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual -Wzero-as-null-pointer-constant -Wnon-virtual-dtor -Ibuild/include -c botan_all.cpp -o build/obj/lib/botan_all.o
In file included from /usr/lib/gcc/x86_64-linux-gnu/7/include/immintrin.h:37:0,
                 from botan_all.cpp:12123:
/usr/lib/gcc/x86_64-linux-gnu/7/include/smmintrin.h: In static member function ‘static void Botan::SHA_256::compress_digest_x86(Botan::secure_vector<unsigned int>&, const uint8_t*, std::size_t)’:
/usr/lib/gcc/x86_64-linux-gnu/7/include/smmintrin.h:166:1: error: inlining failed in call to always_inline ‘__m128i _mm_blend_epi16(__m128i, __m128i, int)’: target specific option mismatch
 _mm_blend_epi16 (__m128i __X, __m128i __Y, const int __M)
 ^~~~~~~~~~~~~~~
botan_all.cpp:52323:11: note: called from here
    STATE1 = _mm_blend_epi16(STATE1, TMP, 0xF0); // CDGH
    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from botan_all.cpp:1738:0:
/usr/lib/gcc/x86_64-linux-gnu/7/include/tmmintrin.h:185:1: error: inlining failed in call to always_inline ‘__m128i _mm_alignr_epi8(__m128i, __m128i, int)’: target specific option mismatch
 _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N)
 ^~~~~~~~~~~~~~~
botan_all.cpp:52322:11: note: called from here
    STATE0 = _mm_alignr_epi8(TMP, STATE1, 8); // ABEF
    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/7/include/immintrin.h:37:0,
                 from botan_all.cpp:12123:
/usr/lib/gcc/x86_64-linux-gnu/7/include/smmintrin.h:166:1: error: inlining failed in call to always_inline ‘__m128i _mm_blend_epi16(__m128i, __m128i, int)’: target specific option mismatch
 _mm_blend_epi16 (__m128i __X, __m128i __Y, const int __M)
 ^~~~~~~~~~~~~~~
botan_all.cpp:52323:11: note: called from here
    STATE1 = _mm_blend_epi16(STATE1, TMP, 0xF0); // CDGH
    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from botan_all.cpp:1738:0:
/usr/lib/gcc/x86_64-linux-gnu/7/include/tmmintrin.h:185:1: error: inlining failed in call to always_inline ‘__m128i _mm_alignr_epi8(__m128i, __m128i, int)’: target specific option mismatch
 _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N)
 ^~~~~~~~~~~~~~~
botan_all.cpp:52322:11: note: called from here
    STATE0 = _mm_alignr_epi8(TMP, STATE1, 8); // ABEF
    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/lib/gcc/x86_64-linux-gnu/7/include/immintrin.h:77:0,
                 from botan_all.cpp:12123:
...
randombit commented 6 years ago

The problem is missing BOTAN_FUNC_ISA declarations to enable ISA extensions on a function by function basis. We are using that elsewhere but it is missing for SHA. Fix coming shortly.

randombit commented 6 years ago

@fgsalomon Thanks for the report. This was introduced in 2.2 when we added support for hardware SHA instructions. It is fixed now in master and so will be fixed in 2.4.0 (being released next week).

Until then you can bypass the issue either by compiling with -msha -msse4.1 -mssse3 (however this will likely result in the compiler emitting SSE 4.1 instructions where not intended, preventing your code from running on older processors), or by disabling the code that calls the SHA instructions with --disable-modules=sha2_32_x86,sha1_x86,shacal2_x86. There is little issue disabling these right now since the only available CPUs that support these instructions are AMD Ryzen and Intel Goldmont.

fgsalomon commented 6 years ago

Wow, that was fast! Thanks! :)