Amalgamated build requires pkcs11.h

[ghost]

Amalgamated build requires pkcs11.h, which, in turn, requires a bunch of other stuff.

Repro (on a recent Intel Xeon):

git clone https://github.com/randombit/botan.git
cd botan
./configure.py --amalgamation --single-amalgamation-file
g++ -std=c++11 botan_all.cpp

Result:

In file included from botan_all.cpp:8:0:
botan_all.h:19302:20: fatal error: pkcs11.h: No such file or directory
compilation terminated.

Expected result:

*.cpp should compile and produce an object file.

[webmaster128]

Why don't you just call make? When you look into the Makefile, you see which include paths you need to add to your compiler call, if you want to call it directly

build/obj/lib/botan_all.o: botan_all.cpp
    $(CXX) $(LIB_FLAGS) -Ibuild/include -Ibuild/include/external -c botan_all.cpp -o $@

[ghost]

Because I thought the whole point of amalgamated build was to create a self contained subset of files that can be compiled anywhere. -Ibuild/include would kind of defeat the purpose.

To quote the docs: "You can also configure Botan to be built using only a single source file; this is quite convenient if you plan to embed the library into another application."

[ghost]

I mean, it's still not a "single file" under the best of circumstances, but even half a dozen would be a lot better for what I'm trying to do than a few hundred.

[webmaster128]

Right. I think -Ibuild/include is not necessary because you have everything from there combined in botan_all.h and botan_all_internal.h. pkcs11 is a special case where headers are used that are not in the library (I don't know why but there is a reason)

./configure.py --amalgamation --single-amalgamation-file --disable-modules=pkcs11
g++ -std=c++11 -c botan_all.cpp

works for me

[ghost]

Maybe it'll work for us as well. Thanks for the suggestion. We need a small subset of ciphers, modes, and hashes. I'll try that and report back later today.

[ghost]

Seems to work fine for us. We used the following flags:

./configure.py --enable-modules=<list of modules> --amalgamation --minimized-build

To reduce the object file size further. I'm closing the issue, feel free to reopen if pkcs11 dependency is unexpected.

[neusdan]

I think it's a bug. The amalgamation build should include the external headers into botan_all.h. What do you think @randombit ?

[randombit]

Bug, especially considering PKCS11 is enabled by default now. Including the external headers directly into botan_all.h does mean that the application cannot include a different version of the PKCS11 headers (say it includes the system version via p11-kit). I'm not sure how much of a problem this would be in practice for anyone, though.

Resolution is either include the headers into botan_all.h, or disable anything with external headers in the amalgamation build. Header inclusion is probably better.

@dmitry-xnor Re "I mean, it's still not a "single file" under the best of circumstances" you can avoid splitting out the C++ code by ISA extensions with --single-amalgamation-file, this requires a sufficiently recent GCC or Clang (one which supports the target function attribute), then there is just botan_all.cpp.

[mgierlings]

I just ran into this bug again with the 3.3.0 release.

git checkout 3.3.0 && ./configure.py --disable-shared --amalgamation && g++ botan_all.cpp
HEAD is now at 9074b04c1 Update for 3.3.0 release
   INFO: ./configure.py invoked with options "--disable-shared --amalgamation"
   INFO: Configuring to build Botan 3.3.0 (revision git:9074b04c1303a24e2084f8325fa570a5ad4f2478)
   INFO: Python version: "3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]"
   INFO: Autodetected platform information: OS="Linux" machine="x86_64" proc="x86_64"
   INFO: Guessing target OS is linux (use --os to set)
   INFO: Guessing to use compiler gcc (use --cc or CXX to set)
   INFO: Guessing target processor is a x86_64 (use --cpu to set)
   INFO: Found rst2man (use --without-rst2man to disable)
   INFO: Using /etc/ssl/certs/ca-certificates.crt as system certificate store
   INFO: Auto-detected compiler version gcc 12.3
   INFO: Auto-detected compiler arch x86_64
   INFO: Target is gcc:12.3-linux-x86_64
   INFO: Assuming target x86_64 is little endian
   INFO: Skipping (dependency failure): asio certstor_sqlite3 sessions_sqlite3
   INFO: Skipping (disabled due to compiler bug): argon2_avx2
   INFO: Skipping (incompatible CPU): aes_armv8 aes_power8 sha1_armv8 sha2_32_armv8 sha2_64_armv8 shacal2_armv8 sm4_armv8
   INFO: Skipping (incompatible OS): certstor_system_macos certstor_system_windows commoncrypto win32_stats
   INFO: Skipping (no enabled compression schemes): compression
   INFO: Skipping (requires external dependency): boost bzip2 lzma sqlite3 tpm zlib
   INFO: Loading modules: adler32 aead aes aes_crystals_xof aes_ni aes_vperm argon2 argon2_ssse3 argon2fmt aria asn1 auto_rng base base32 base58 base64 bcrypt bcrypt_pbkdf bigint blake2 blake2mac blake2s block blowfish camellia cascade cast128 cbc ccm certstor_flatfile certstor_sql certstor_system cfb chacha chacha20poly1305 chacha_avx2 chacha_avx512 chacha_rng chacha_simd32 checksum cmac comb4p cpuid crc24 crc32 cryptobox cshake_xof ctr curve25519 des dh dilithium dilithium_aes dilithium_common dl_algo dl_group dlies dsa dyn_load eax ec_group ec_h2c ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 frodokem frodokem_aes frodokem_common gcm getentropy ghash ghash_cpu ghash_vperm gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea idea_sse2 iso9796 kdf kdf1 kdf1_iso18033 kdf2 keccak keccak_perm keccak_perm_bmi2 keypair kmac kuznyechik kyber kyber_90s kyber_common lion locking_allocator mac mce md4 md5 mdx_hash mem_pool mgf1 mode_pad modes mp nist_keywrap noekeon noekeon_simd numbertheory ocb ofb par_hash passhash9 pbes2 pbkdf pbkdf2 pem pgp_s2k pk_pad pkcs11 poly1305 poly_dbl prf_tls prf_x942 processor_rng psk_db pubkey raw_hash rc4 rdseed rfc3394 rfc6979 rmd160 rng roughtime rsa salsa20 scrypt seed serpent serpent_avx2 serpent_avx512 serpent_simd sessions_sql sha1 sha1_sse2 sha1_x86 sha2_32 sha2_32_bmi2 sha2_32_x86 sha2_64 sha2_64_bmi2 sha3 shacal2 shacal2_avx2 shacal2_simd shacal2_x86 shake shake_cipher shake_xof simd simd_avx2 simd_avx512 siphash siv skein sm2 sm3 sm4 socket sodium sp800_108 sp800_56a sp800_56c sphincsplus_common sphincsplus_sha2 sphincsplus_shake srp6 stateful_rng stream streebog system_rng thread_utils threefish_512 tls tls12 tls13 tls13_pqc tls_cbc trunc_hash tss twofish utils uuid whirlpool x509 x919_mac xmss xof xts zfec zfec_sse2 zfec_vperm
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Writing amalgamation header to botan_all.h
   INFO: Writing amalgamation source to botan_all.cpp
   INFO: Botan 3.3.0 (revision git:9074b04c1303a24e2084f8325fa570a5ad4f2478) (unreleased undated) build setup is complete
In file included from botan_all.cpp:8:
botan_all.h:19302:10: fatal error: pkcs11.h: No such file or directory
19302 | #include "pkcs11.h"
      |          ^~~~~~~~~~
compilation terminated.