randombit / botan

Cryptography Toolkit
https://botan.randombit.net
BSD 2-Clause "Simplified" License
2.53k stars 562 forks source link

'++crypto' is not a recognized feature for this target (ignoring feature) #3487

Open oviano opened 1 year ago

oviano commented 1 year ago

I get this warning together with a similar one about "++simd" on my macOS arm64 host when building native macOS, arm64 iOS or arm64 Android.

I also get a similar warning about when building Android armv7 about "+fpu=neon".

Here's an example with iOS arn64:

python3 configure.py --minimized --enable-modules=tls12,tls13,sqlite3,sessions_sqlite3,srp6,ed25519,curve25519,system_rng,bcrypt,crc32,hkdf,chacha20poly1305,nist_keywrap,ffi,aes_armv8,aes_ni,aes_power8,aes_vperm,idea_sse2,serpent_avx2,shacal2_armv8,shacal2_avx2,shacal2_x86,sm4_armv8,rdseed,sha1_armv8,sha1_sse2,sha1_x86,sha2_32_armv8,sha2_32_bmi2,sha2_32_x86,sha2_64_bmi2,sha3_bmi2,zfec_sse2,zfec_vperm,argon2_avx2,argon2_ssse3,processor_rng,chacha_avx2,ghash_cpu,ghash_vperm,simd,simd_avx2 --os=ios --cpu=arm64 --cc=clang --cc-min-version=13.0 --cc-abi-flags="-arch arm64" --disable-shared
   INFO: configure.py invoked with options "--minimized --enable-modules=tls12,tls13,sqlite3,sessions_sqlite3,srp6,ed25519,curve25519,system_rng,bcrypt,crc32,hkdf,chacha20poly1305,nist_keywrap,ffi,aes_armv8,aes_ni,aes_power8,aes_vperm,idea_sse2,serpent_avx2,shacal2_armv8,shacal2_avx2,shacal2_x86,sm4_armv8,rdseed,sha1_armv8,sha1_sse2,sha1_x86,sha2_32_armv8,sha2_32_bmi2,sha2_32_x86,sha2_64_bmi2,sha3_bmi2,zfec_sse2,zfec_vperm,argon2_avx2,argon2_ssse3,processor_rng,chacha_avx2,ghash_cpu,ghash_vperm,simd,simd_avx2 --os=ios --cpu=arm64 --cc=clang --cc-min-version=13.0 --cc-abi-flags=-arch arm64 --disable-shared"
   INFO: Configuring to build Botan 3.0.0-rc1 (revision git:604672a06366b7ddb94632d1027c546c49a67caa)
   INFO: Running under 3.9.6 (default, Mar 10 2023, 20:16:38) [Clang 14.0.3 (clang-1403.0.22.14.1)]
   INFO: Autodetected platform information: OS="Darwin" machine="arm64" proc="arm"
   INFO: Using /etc/ssl/cert.pem as system certificate store
   INFO: Auto-detected compiler arch arm64
   INFO: Target is clang:13.0-ios-arm64
   INFO: Assuming target arm64 is little endian
   INFO: Skipping (incompatible CPU): aes_ni aes_power8 argon2_avx2 argon2_ssse3 chacha_avx2 ghash_vperm idea_sse2 processor_rng rdseed serpent_avx2 sha1_sse2 sha1_x86 sha2_32_bmi2 sha2_32_x86 sha2_64_bmi2 sha3_bmi2 shacal2_avx2 shacal2_x86 simd_avx2 zfec_sse2
   INFO: Skipping (incompatible OS): certstor_system_macos certstor_system_windows getentropy win32_stats
   INFO: Skipping (incompatible compiler): sm4_armv8
   INFO: Skipping (not requested): adler32 argon2 argon2fmt aria asio base32 base58 bcrypt_pbkdf blake2 blake2mac camellia cascade cast128 cbc ccm certstor_flatfile certstor_sql certstor_sqlite3 certstor_system cfb chacha_rng chacha_simd32 cmac comb4p compression crc24 cryptobox des dilithium dilithium_aes dilithium_common dlies dsa dyn_load eax ec_h2c ecgdsa ecies eckcdsa elgamal eme_oaep eme_raw emsa_raw emsa_x931 fd_unix filters fpe_fe1 gmac gost_28147 gost_3410 gost_3411 hotp http_util idea iso9796 kdf1 kdf1_iso18033 kdf2 keccak kyber kyber_90s kyber_common lion locking_allocator mce md4 md5 mem_pool mode_pad noekeon noekeon_simd ocb ofb par_hash passhash9 pbes2 pgp_s2k pkcs11 poly_dbl prf_x942 psk_db raw_hash rc4 rfc3394 rfc6979 rmd160 roughtime salsa20 scrypt seed serpent serpent_simd sha3 shacal2_simd shake shake_cipher siphash siv skein sm2 sm3 sm4 socket sodium sp800_108 sp800_56a sp800_56c streebog thread_utils threefish_512 tls_cbc trunc_hash tss twofish uuid whirlpool x919_mac xmss xts
   INFO: Skipping (requires external dependency): boost bzip2 commoncrypto lzma tpm zlib
   INFO: Enabling use of external dependency sqlite3
   INFO: Loading modules: aead aes aes_armv8 aes_vperm asn1 auto_rng base base64 bcrypt bigint block blowfish chacha chacha20poly1305 checksum cpuid crc32 ctr curve25519 dh dl_algo dl_group ec_group ecc_key ecdh ecdsa ed25519 eme_pkcs1 emsa_pkcs1 emsa_pssr entropy ffi gcm ghash ghash_cpu hash hash_id hex hkdf hmac hmac_drbg kdf keypair mac mdx_hash mgf1 modes mp nist_keywrap numbertheory pbkdf pbkdf2 pem pk_pad poly1305 prf_tls pubkey rng rsa sessions_sql sessions_sqlite3 sha1 sha1_armv8 sha2_32 sha2_32_armv8 sha2_64 shacal2 shacal2_armv8 simd sqlite3 srp6 stateful_rng stream system_rng tls tls12 tls13 utils x509 zfec zfec_vperm
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Botan 3.0.0-rc1 (revision git:604672a06366b7ddb94632d1027c546c49a67caa) (unreleased undated) build setup is complete

Then I get output like this in places:

clang++  -fstack-protector -pthread -stdlib=libc++ -arch arm64 -std=c++20 -D_REENTRANT  -O3 -DBOTAN_IS_BEING_BUILT -Wall -Wextra -Wpedantic -Wshadow -Wstrict-aliasing -Wstrict-overflow=5 -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual -Wshorten-64-to-32 -Wcomma -Wdocumentation -march=armv8+crypto -I build/include -c src/lib/block/aes/aes_armv8/aes_armv8.cpp -o build/obj/lib/block_aes_armv8.o
'++crypto' is not a recognized feature for this target (ignoring feature)
'++crypto' is not a recognized feature for this target (ignoring feature)

If I change the "+crypto" to "+nocrypto" in the above, then I get a bunch of compile errors, which makes sense:

clang++  -fstack-protector -pthread -stdlib=libc++ -arch arm64 -std=c++20 -D_REENTRANT  -O3 -DBOTAN_IS_BEING_BUILT -Wall -Wextra -Wpedantic -Wshadow -Wstrict-aliasing -Wstrict-overflow=5 -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual -Wshorten-64-to-32 -Wcomma -Wdocumentation -march=armv8+nocrypto -I build/include -c src/lib/block/aes/aes_armv8/aes_armv8.cpp -o build/obj/lib/block_aes_armv8.o
src/lib/block/aes/aes_armv8/aes_armv8.cpp:80:7: error: use of undeclared identifier 'vaeseq_u8'
      AES_ENC_4_ROUNDS(K0);
      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:20:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B0 = vaesmcq_u8(vaeseq_u8(B0, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:80:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:21:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B1 = vaesmcq_u8(vaeseq_u8(B1, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:80:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:22:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B2 = vaesmcq_u8(vaeseq_u8(B2, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:80:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:23:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B3 = vaesmcq_u8(vaeseq_u8(B3, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:81:7: error: use of undeclared identifier 'vaeseq_u8'
      AES_ENC_4_ROUNDS(K1);
      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:20:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B0 = vaesmcq_u8(vaeseq_u8(B0, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:81:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:21:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B1 = vaesmcq_u8(vaeseq_u8(B1, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:81:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:22:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B2 = vaesmcq_u8(vaeseq_u8(B2, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:81:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:23:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B3 = vaesmcq_u8(vaeseq_u8(B3, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:82:7: error: use of undeclared identifier 'vaeseq_u8'
      AES_ENC_4_ROUNDS(K2);
      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:20:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B0 = vaesmcq_u8(vaeseq_u8(B0, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:82:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:21:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B1 = vaesmcq_u8(vaeseq_u8(B1, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:82:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:22:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B2 = vaesmcq_u8(vaeseq_u8(B2, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:82:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:23:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B3 = vaesmcq_u8(vaeseq_u8(B3, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:83:7: error: use of undeclared identifier 'vaeseq_u8'
      AES_ENC_4_ROUNDS(K3);
      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:20:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B0 = vaesmcq_u8(vaeseq_u8(B0, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:83:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:21:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B1 = vaesmcq_u8(vaeseq_u8(B1, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:83:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:22:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B2 = vaesmcq_u8(vaeseq_u8(B2, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:83:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:23:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B3 = vaesmcq_u8(vaeseq_u8(B3, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:84:7: error: use of undeclared identifier 'vaeseq_u8'
      AES_ENC_4_ROUNDS(K4);
      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:20:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B0 = vaesmcq_u8(vaeseq_u8(B0, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:84:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:21:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B1 = vaesmcq_u8(vaeseq_u8(B1, K));   \
                      ^
src/lib/block/aes/aes_armv8/aes_armv8.cpp:84:7: error: use of undeclared identifier 'vaeseq_u8'
src/lib/block/aes/aes_armv8/aes_armv8.cpp:22:23: note: expanded from macro 'AES_ENC_4_ROUNDS'
      B2 = vaesmcq_u8(vaeseq_u8(B2, K));   \
                      ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

So my question really is do these warnings matter?

randombit commented 1 year ago

This is only seen on the Apple version of clang. I'm not sure what the problem is; possibly it's simply because for Apple hardware, all aarch64 hardware in existence supports the crypto extension, so explicitly marking support for it is "pointless". But of course doing so is required for standard Clang, or with GCC. The warnings are afaik harmless.

oviano commented 1 year ago

I've had a bit of a closer look at this.

It seems it might relate to a difference between the way function attribute(target) is treated on gcc vs clang:

https://gcc.gnu.org/onlinedocs/gcc-7.5.0/gcc/AArch64-Function-Attributes.html https://clang.llvm.org/docs/AttributeReference.html#target

I have found that:

  1. With my compiler (Apple clang 14), the subtarget features are not recognised if they have a preceding +. I assume the compiler is adding the + implicitly, which explains why the warning shows ++. Removing the + preceding "crypto" throughout Botan removes this warning and it is now recognised by my compiler.
  2. Consider this block of code:
#if defined(BOTAN_SIMD_USE_SSE2)
  #define BOTAN_SIMD_ISA "sse2"
  #define BOTAN_VPERM_ISA "ssse3"
  #define BOTAN_CLMUL_ISA "pclmul"
#elif defined(BOTAN_SIMD_USE_NEON)
  #if defined(BOTAN_TARGET_ARCH_IS_ARM64)
    #define BOTAN_SIMD_ISA "+simd"
    #define BOTAN_CLMUL_ISA "+crypto"
  #else
    #define BOTAN_SIMD_ISA "fpu=neon"
  #endif
  #define BOTAN_VPERM_ISA BOTAN_SIMD_ISA
#elif defined(BOTAN_SIMD_USE_ALTIVEC)
  #define BOTAN_SIMD_ISA "altivec"
  #define BOTAN_VPERM_ISA "altivec"
  #define BOTAN_CLMUL_ISA "crypto"
#endif

Removing the + preceding "simd" still doesn't get it recognised with my compiler.

However, according to this page:

https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html

Specifically this excerpt:

‘+simd’
The Advanced SIMD (Neon) v1 and the VFPv3 floating-point instructions. The extensions ‘+neon’ and ‘+neon-vfpv3’ can be used as aliases for this extension.

It turns out that if I replace the "+simd" with "neon" it is recognised and removes the warning.

Finally, the line #define BOTAN_SIMD_ISA "fpu=neon" which presumably derives from the command line option -mfpu=neon is not mentioned by either gcc or clang as an option that can be specified via a function attribute so perhaps that should just be "+simd" (gcc) or "neon" (clang) too?

Essentially if I change all "+crypo" to "crypto", "+simd" to "neon" and "fpu=neon" to "neon" then it builds without warnings on all platforms (macOS, iOS, Android) with my clang.

So maybe it just needs tweaking slightly for clang vs gcc?

Of course, maybe this is all irrelevant if the code generated is the same, but I'm a pedant and wanted to understand where it was coming from.

randombit commented 1 year ago

Thanks for looking into this. I was hoping we could just use crypto everywhere but GCC won't have it

error: arch extension 'crypto' should be prefixed by '+'

oviano commented 1 year ago

Does the "fpu=neon" cause any warnings on gcc?

Also, I don't know if it's only Apple's clang that doesn't like the preceding "+", or whether it's clang on all/other platforms, and whether it varies according to the version.

randombit commented 1 year ago

Does the "fpu=neon" cause any warnings on gcc?

No. I just checked and GCC 12 targeting arm32 will accept fpu=neon, or +simd (I suspect that is new behavior), but not neon or simd.

randombit commented 1 year ago

Looks like broadly LLVM is interested in being compatible with GCC, but isn't yet https://github.com/llvm/llvm-project/issues/56480

oviano commented 1 year ago

Ok. I'm trying to find a definite source that describes how function target attributes work.

This seems like the one for gcc

https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html

Haven't found the equivalent for clang yet.

EDIT: probably this is the one, but it's fairly vague.

https://clang.llvm.org/docs/AttributeReference.html#target

oviano commented 1 year ago

Ok, I ran some tests on Godbolt and it seems that from clang 16 onwards) there will be greater compatibility at least for armv8-a. It seems that the compatibility fixes haven't made it to armv7-a trunk.

https://godbolt.org/z/56bE9o7dh

armv7-a clang (trunk) targeting armv7-a

+crypto     FAIL (expected)
+simd       FAIL
+neon       FAIL
crypto      OK (unexpected?)
simd        FAIL
neon        OK
fpu=neon    FAIL

arm gcc 12.2 targeting armv7-a

+crypto     FAIL (expected)
+simd       OK
+neon       OK
crypto      FAIL (expected)
simd        FAIL
neon        FAIL
fpu=neon    OK

armv8-a clang (trunk) targeting armv8-a

+crypto     OK
+simd       OK
+neon       OK
crypto      OK
simd        OK
neon        OK
fpu=neon    FAIL (expected)

ARM64 gcc trunk targeting amv8-a

+crypto     OK
+simd       OK
+neon       FAIL
crypto      FAIL
simd        FAIL
neon        FAIL
fpu=neon    FAIL (expected)

There is still not going to be a one-size-fits-all solution though.

I suggest something like this:

#if defined(BOTAN_SIMD_USE_SSE2)
  #define BOTAN_SIMD_ISA "sse2"
  #define BOTAN_VPERM_ISA "ssse3"
  #define BOTAN_CLMUL_ISA "pclmul"
#elif defined(BOTAN_SIMD_USE_NEON)
  #if defined(BOTAN_TARGET_ARCH_IS_ARM64)
      #if defined(BOTAN_BUILD_COMPILER_IS_CLANG)
        #define BOTAN_SIMD_ISA "neon"
        #define BOTAN_CLMUL_ISA "crypto"
      #else
        #define BOTAN_SIMD_ISA "+simd"
        #define BOTAN_CLMUL_ISA "+crypto"
      #endif
  #else
    #if defined(BOTAN_BUILD_COMPILER_IS_CLANG)
      #define BOTAN_SIMD_ISA "neon"
    #else
      #define BOTAN_SIMD_ISA "fpu=neon"
    #endif
  #endif
  #define BOTAN_VPERM_ISA BOTAN_SIMD_ISA

and elsewhere

if defined(BOTAN_BUILD_COMPILER_IS_CLANG)
BOTAN_FUNC_ISA("crypto")
#else
BOTAN_FUNC_ISA("+crypto")
#endif