search

randombit / botan

Cryptography Toolkit
https://botan.randombit.net
BSD 2-Clause "Simplified" License
2.59k stars 567 forks source link

Build fails on PowerPC 64 with clang #2547

Closed guidovranken closed 3 years ago

guidovranken commented 3 years ago

This is how I build Botan: https://github.com/guidovranken/cryptofuzz/blob/master/docs/botan.md

I'm using an Ubuntu 20 shell from https://openpower.ic.unicamp.br/ for PowerPC emulation.

clang++-11  -fstack-protector -pthread -fsanitize=fuzzer-no-link -g -O3 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -I /home/ubuntu/cf/openssl/include -std=c++17 -D_REENTRANT  -fsanitize=fuzzer-no-link -g -O3 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -I /home/ubuntu/cf/openssl/include -DBOTAN_IS_BEING_BUILT -Wall -Wextra -Wpedantic -Wshadow -Wstrict-aliasing -Wstrict-overflow=5 -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual -Wshorten-64-to-32 -maltivec -mcrypto -Ibuild/include -Ibuild/include/external -c src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp -o build/obj/lib/utils_ghash_cpu.o
src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp:74:48: warning: Use of 'long' with '__vector' is deprecated [-Wdeprecated]
            reinterpret_cast<__vector unsigned long>(i1.raw()),
                                               ^
src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp:75:48: warning: Use of 'long' with '__vector' is deprecated [-Wdeprecated]
            reinterpret_cast<__vector unsigned long>(i2.raw())))
                                               ^
src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp:73:10: error: use of undeclared identifier '__builtin_crypto_vpmsumd'; did you mean '__builtin_crypto_vpmsumb'?
         __builtin_crypto_vpmsumd(
         ^~~~~~~~~~~~~~~~~~~~~~~~
         __builtin_crypto_vpmsumb
/usr/lib/llvm-11/lib/clang/11.0.0/include/altivec.h:16177:1: note: '__builtin_crypto_vpmsumb' declared here
__builtin_crypto_vpmsumb(vector unsigned long long __a,
^
2 warnings and 1 error generated.
make: *** [Makefile:1086: build/obj/lib/utils_ghash_cpu.o] Error 1
randombit commented 3 years ago

What version of Clang is this? And, are you sure this is building for 64-bit PPC, and not 32-bit?

randombit commented 3 years ago

Q1: I missed the clang-11 invocation. Q2: This is being built for 32-bit POWER - on ppc64 there would be an -m64 between clang++-11 and -fstack-protector. Can you post configure.pys arguments and output?

randombit commented 3 years ago

I think this is a bug in configure.py. Are you configuring as --cpu=ppc64le? If so try --cpu=ppc64 --with-endian=little

randombit commented 3 years ago

Alternately can you try this

diff --git a/configure.py b/configure.py
index 4ed480722..71856e5a5 100755
--- a/configure.py
+++ b/configure.py
@@ -969,7 +969,7 @@ class ModuleInfo(InfoObject):

     def compatible_cpu(self, archinfo, options):
         arch_name = archinfo.basename
-        cpu_name = options.cpu
+        cpu_name = options.arch

         if self.endian != 'any':
             if self.endian != options.with_endian:
@@ -1330,7 +1330,7 @@ class CompilerInfo(InfoObject): # pylint: disable=too-many-instance-attributes
                     yield all_except

             yield options.os
-            yield options.cpu
+            yield options.arch

         abi_link = set()
         for what in mach_abi_groups():
guidovranken commented 3 years ago

Neither of your proposed solutions work.

/proc/cpuinfo:

processor   : 0
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 1
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 2
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 3
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 4
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 5
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 6
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

processor   : 7
cpu     : POWER8 (architected), altivec supported
clock       : 3425.000000MHz
revision    : 2.1 (pvr 004b 0201)

timebase    : 512000000
platform    : pSeries
model       : IBM pSeries (emulated by qemu)
machine     : CHRP IBM pSeries (emulated by qemu)
MMU     : Hash

configure.py output:

$ ./configure.py --cc-bin=$CXX --cc-abi-flags="$CXXFLAGS" --disable-shared --disable-modules=locking_allocator --build-targets=static --without-documentation
   INFO: ./configure.py invoked with options "--cc-bin=clang++-11 --cc-abi-flags= --disable-shared --disable-modules=locking_allocator --build-targets=static --without-documentation"
   INFO: Configuring to build Botan 3.0.0-alpha0 (revision git:972266811936d5e32860302542bfadc1c0d3e553)
   INFO: Running under 2.7.18 (default, Aug 24 2020, 19:12:23) [GCC 10.2.0]
   INFO: Autodetected platform information: OS="Linux" machine="ppc64le" proc="ppc64le"
   INFO: Guessing target OS is linux (use --os to set)
   INFO: Guessing target processor is a ppc64 (use --cpu to set)
   INFO: Using /etc/ssl/certs/ca-certificates.crt as system certificate store
   INFO: Canonicalized CPU target ppc64le to ppc64
   INFO: Auto-detected compiler version 11.0
   INFO: Auto-detected compiler arch ppc64
   INFO: Target is clang:11.0-linux-ppc64
   INFO: Skipping (dependency failure): asio certstor_sqlite3 sessions_sqlite3
   INFO: Skipping (disabled by user): locking_allocator
   INFO: Skipping (incompatible CPU): aes_armv8 aes_ni chacha_avx2 ghash_vperm idea_sse2 rdseed serpent_avx2 sha1_armv8 sha1_sse2 sha1_x86 sha2_32_armv8 sha2_32_bmi2 sha2_32_x86 sha2_64_bmi2 sha3_bmi2 shacal2_avx2 shacal2_x86 simd_avx2 sm4_armv8 threefish_512_avx2
   INFO: Skipping (incompatible OS): certstor_system_macos certstor_system_windows commoncrypto getentropy win32_stats
   INFO: Skipping (no enabled compression schemes): compression
   INFO: Skipping (requires external dependency): boost bzip2 lzma openssl sqlite3 tpm zlib
   INFO: Loading modules: adler32 aead aes aes_power8 aes_vperm argon2 aria asn1 auto_rng base base32 base58 base64 bcrypt bcrypt_pbkdf bigint blake2 blake2mac block blowfish camellia cascade cast128 cbc ccm cecpq1 certstor_flatfile certstor_sql certstor_system cfb chacha chacha20poly1305 chacha_rng chacha_simd32 checksum cmac comb4p cpuid crc24 crc32 cryptobox ctr curve25519 des dev_random dh dl_algo dl_group dlies dsa dyn_load eax ec_group ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 gcm ghash ghash_cpu gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea iso9796 kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion mac mce md4 md5 mdx_hash mem_pool mgf1 mode_pad modes mp newhope nist_keywrap noekeon noekeon_simd numbertheory ocb ofb par_hash passhash9 pbes2 pbkdf pbkdf2 pem pgp_s2k pk_pad pkcs11 poly1305 poly_dbl prf_tls prf_x942 proc_walk processor_rng psk_db pubkey rc4 rfc3394 rfc6979 rmd160 rng roughtime rsa salsa20 scrypt seed serpent serpent_simd sessions_sql sha1 sha2_32 sha2_64 sha3 shacal2 shacal2_simd shake shake_cipher simd siphash siv skein sm2 sm3 sm4 socket sodium sp800_108 sp800_56a sp800_56c srp6 stateful_rng stream streebog system_rng thread_utils threefish_512 tls tls_10 tls_cbc tss twofish utils uuid whirlpool x509 x919_mac xmss xts
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Botan 3.0.0-alpha0 (revision git:972266811936d5e32860302542bfadc1c0d3e553) (unreleased undated) build setup is complete

What I did was change the code to

          __builtin_crypto_vpmsumb(
             reinterpret_cast<__vector unsigned int>(i1.raw()),
             reinterpret_cast<__vector unsigned int>(i2.raw())))

Maybe it's a Clang issue? See https://github.com/antonblanchard/crc32-vpmsum/blob/master/clang_workaround.h

randombit commented 3 years ago

I think it is a Clang specific thing (we build/test using GCC on ppc64 in CI and I also test on the GCC compile farm POWER8/POWER9 machines, no problems there). See https://github.com/weidai11/cryptopp/blob/master/ppc_simd.h#L2490

randombit commented 3 years ago

Can you try this? Sorry I cannot test this as the Clangs on the GCC compile farm are really old and I do not feel like bootstrapping LLVM.

diff --git a/src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp b/src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp
index f60dfbd38..d51c2bf9d 100644
--- a/src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp
+++ b/src/lib/utils/ghash/ghash_cpu/ghash_cpu.cpp
@@ -70,7 +70,7 @@ BOTAN_FORCE_INLINE SIMD_4x32 BOTAN_FUNC_ISA(BOTAN_CLMUL_ISA) clmul(const SIMD_4x

    return SIMD_4x32(
       reinterpret_cast<__vector unsigned int>(
-         __builtin_crypto_vpmsumd(
+         __builtin_altivec_crypto_vpmsumd(
             reinterpret_cast<__vector unsigned long>(i1.raw()),
             reinterpret_cast<__vector unsigned long>(i2.raw())))
       );
guidovranken commented 3 years ago

I've just confirmed that this works.

randombit commented 3 years ago

Fixed on master and release-2 branches now. Thanks for the report and the help testing the fix.