randombit / botan

Cryptography Toolkit
https://botan.randombit.net
BSD 2-Clause "Simplified" License
2.55k stars 561 forks source link

Two questions regarding a custom build #1416

Closed DevSidious closed 6 years ago

DevSidious commented 6 years ago

Premises:

I need to generate a custom build of Botan, aimed at X509 certificate manipulation. I want to load, inspect and validate existing certificates, as well as generate and save new ones. Of utmost importnace is the ability to support both RSA and ECC keys.

After fiddling with configure.py for a while I reached this combination of flags:

C:\Git\botan>configure.py --cc=msvc --os=windows --minimized-build --enable-modules=x509,ecdsa,rsa --amalgamation --single-amalgamation-file --prefix="C:\Botan"

INFO: Skipping (incompatible OS): darwin_secrandom dev_random fd_unix getentropy proc_walk

INFO: Skipping (incompatible compiler): aes_armv8 pmull sha1_armv8 sha1_x86 sha2_32_armv8 sha2_32_x86 shacal2_x86

INFO: Skipping (not requested): adler32 aead aes aes_ni aes_ssse3 aont aria auto_rng bcrypt blake2 blowfish camellia cascade cast cbc_mac ccm cecpq1 certstor_sql certstor_sqlite3 cfb chacha chacha20poly1305 chacha_rng chacha_sse2 clmul clmul_ssse3 cmac codec_filt comb4p compression crc24 crc32 cryptobox ctr curve25519 des dh dl_algo dl_group dlies dsa dyn_load eax ecdh ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa_pkcs1 emsa_raw emsa_x931 ffi filters fpe_fe1 gcm gmac gost_28147 gost_3410 gost_3411 hash_id hkdf hmac_drbg hotp http_util idea idea_sse2 iso9796 kasumi kdf1 kdf1_iso18033 kdf2 keccak lion locking_allocator mce mceies md4 md5 misty1 newhope nist_keywrap noekeon noekeon_simd ocb ofb par_hash passhash9 pbkdf1 pgp_s2k pkcs11 poly1305 poly_dbl prf_tls prf_x942 psk_db rc4 rdrand rdrand_rng rdseed rfc3394 rfc6979 rmd160 salsa20 seed serpent serpent_simd sessions_sql sessions_sqlite3 sha1_sse2 sha2_64 sha3 shacal2 shacal2_simd shake shake_cipher simd siphash siv skein sm2 sm3 sm4 socket sp800_108 sp800_56a sp800_56c srp6 stateful_rng streebog system_rng threefish threefish_avx2 tiger tls tls_cbc tss twofish whirlpool win32_stats x919_mac xmss xtea xts

INFO: Skipping (requires external dependency): bearssl boost bzip2 lzma openssl sqlite3 tpm zlib

INFO: Loading modules: asn1 base base64 bigint block cbc cpuid ec_gfp ec_group ecc_key ecdsa emsa1 emsa_pssr entropy hash hex hmac kdf keypair mac mdx_hash mgf1 mode_pad modes mp numbertheory pbes2 pbkdf pbkdf2 pem pk_pad pubkey rng rsa sha1 sha2_32 stream utils x509

The questions:

  1. Given the list of modules above, is there anything I am missing in order to fulfill my purpose?
  2. Given the list of modules above, is there anything else I could further strip out? I suspect that modules such as block and cbc could be stripped out, among others.

Any input would be greatly appreciated.

Thank you.

PS: congratulations to the authors for this excellent library.

randombit commented 6 years ago
  1. For RSA you will almost certainly want to also enable emsa_pkcs1 since in practice almost all X.509 RSA certificates are signed with that format. You might want to also enable SHA-384/SHA-512 (sha2_64) since those are sometimes used in ECDSA certificates.

  2. Unfortunately it's not possible to further disable beyond what is being configured above. A lot of what is being pulled in (cbc, block, pbkdf2 etc) is because the public key code depends on pbes2, which is the encryption scheme used to encrypt private keys. PBES2 in turn pulls in PBKDF2, CBC, etc. If you are going to encrypt private keys, PBES2 is required. However, if you are not it is kind of a bunch of dead weight. It turns out to be easy to make PBES2 optional, and just disable encryption/decryption of private keys in that case. (#1417)

DevSidious commented 6 years ago

Sounds like pbes2 is something I want to keep. Flags for configure.py now become:

--cc=msvc --os=windows --minimized-build --enable-modules=x509,ecdsa,rsa,emsa_pkcs1,sha2_64

Resulting in:

INFO: Loading modules: asn1 base base64 bigint block cbc cpuid ec_gfp ec_group ecc_key ecdsa emsa1 emsa_pkcs1 emsa_pssr entropy hash hash_id hex hmac kdf keypair mac mdx_hash mgf1 mode_pad modes mp numbertheory pbes2 pbkdf pbkdf2 pem pk_pad pubkey rng rsa sha1 sha2_32 sha2_64 stream utils x509

Excellent! Big thank you @randombit .

randombit commented 6 years ago

OK, if you want to use PBES2 you should probably also enable at least one cipher (AES being the best choice for interop).

DevSidious commented 6 years ago

@randombit
And if I build for x86_64 I should probably also enable aes_ni and aes_ssse3, correct ?

randombit commented 6 years ago

Yes. These days you can probably get away with skipping the SSSE3 code since AES-NI is pretty common but having the SSSE3 code around won't hurt unless you really need to shrink binaries to the minimum.

DevSidious commented 6 years ago

Building with configure.py --cc=msvc --os=windows --minimized-build --enable-modules=x509,ecdsa,rsa,emsa_pkcs1,sha2_64,aes,aes_ni,aes_ssse3 and testing with botan-test.exe results in botan-test.exe crashing at rsa_sign: stage.

Output below:

Testing Botan 2.4.0 (unreleased, revision git:da70e06c13c57245d4e79122e2ccefe08ccfa20d, distribution unspecified)
Starting tests drbg_seed:0000C33CD0426888
block:
AES-128 ran 15054 tests in 29.90 msec all ok
AES-192 ran 17550 tests in 35.32 msec all ok
AES-256 ran 20046 tests in 40.21 msec all ok
stream:
hash:
SHA-160 ran 1340 tests in 1.16 msec all ok
SHA-224 ran 14 tests in 0.02 msec all ok
SHA-256 ran 3494 tests in 3.22 msec all ok
SHA-384 ran 57 tests in 0.06 msec all ok
SHA-512 ran 57 tests in 0.06 msec all ok
SHA-512-256 ran 7 tests in 0.01 msec all ok
mac:
HMAC(SHA-160) ran 66 tests in 0.25 msec all ok
HMAC(SHA-256) ran 55 tests in 0.19 msec all ok
modes:
AES-128/CBC/CTS ran 66 tests in 0.22 msec all ok
AES-128/CBC/NoPadding ran 22 tests in 0.11 msec all ok
AES-192/CBC/NoPadding ran 11 tests in 0.05 msec all ok
AES-256/CBC/NoPadding ran 11 tests in 0.04 msec all ok
aead:
kdf:
pbkdf:
PBKDF2(HMAC(SHA-160)) ran 20 tests in 32.68 msec all ok
PBKDF2(HMAC(SHA-256)) ran 2 tests in 25.56 msec all ok
PBKDF2(HMAC(SHA-384)) ran 2 tests in 15.90 msec all ok
PBKDF2(HMAC(SHA-512)) ran 2 tests in 15.83 msec all ok
hmac_drbg:
util:
CT utils ran 20 tests all ok
Util load/store ran 229 tests all ok
Util round_down ran 6 tests in 0.01 msec all ok
Util round_up ran 11 tests in 0.15 msec all ok
asn1:
ASN.1 ASCII encoding ran 2 tests all ok
ASN.1 ASCII parsing ran 1 tests all ok
ASN.1 BMP string (UCS-2) parsing ran 1 tests all ok
ASN.1 UTF-8 encoding ran 2 tests all ok
ASN.1 UTF-8 parsing ran 1 tests all ok
ASN.1 universal string (UCS-4) parsing ran 1 tests all ok
BER nested indefinite length ran 1 tests all ok
BER stack recursion ran 1 tests all ok
asn1_printer:
ASN1_Pretty_Printer ran 5 tests all ok
base64:
Base64 ran 64 tests in 0.08 msec all ok
bc_pad:
ESP ran 15 tests in 0.01 msec all ok
NoPadding ran 10 tests in 0.01 msec all ok
OneAndZeros ran 13 tests in 0.01 msec all ok
PKCS7 ran 12 tests in 0.01 msec all ok
X9.23 ran 13 tests in 0.01 msec all ok
bigint_unit:
BigInt IO operators ran 8 tests all ok
BigInt encoding functions ran 1 tests all ok
BigInt prime generation ran 151 tests all ok
BigInt size functions ran 72 tests all ok
BigInt::random_integer ran 15010 tests in 4.45 msec all ok
bn_add:
BigInt Addition ran 292 tests in 96.54 msec all ok
bn_div:
BigInt Divide ran 76 tests in 5.94 msec all ok
bn_gcd:
BigInt Mod ran 1 tests in 0.02 msec all ok
bn_invmod:
BigInt InvMod ran 305 tests in 64.95 msec all ok
bn_isprime:
BigInt Test NonPrime ran 107 tests in 254.93 msec all ok
BigInt Test Prime ran 17 tests in 7.48 msec all ok
bn_lshift:
BigInt Lshift ran 100 tests in 44.88 msec all ok
bn_mod:
BigInt Mod ran 171 tests in 9.45 msec all ok
bn_mul:
BigInt Multiply ran 416 tests in 312.15 msec all ok
bn_powmod:
BigInt Powmod ran 213 tests in 302.42 msec all ok
bn_ressol:
BigInt Ressol ran 22 tests in 9.78 msec all ok
bn_rshift:
BigInt Rshift ran 102 tests in 10.74 msec all ok
bn_sqr:
BigInt Square ran 42 tests in 5.36 msec all ok
bn_sub:
BigInt Subtraction ran 126 tests in 28.46 msec all ok
charset:
Charset ran 14 tests in 0.04 msec all ok
Charset negative tests ran 5 tests all ok
cpuid:
CPUID ran 6 tests all ok
dsa_param:
DSA Parameter Generation ran 10 tests in 982.71 msec all ok
ecc_invalid:
ECC invalid keys ran 5 tests in 2.37 msec all ok
ecc_pointmul:
ECC Scalarmult frp256v1 ran 104 tests in 121.94 msec all ok
ECC Scalarmult secp192r1 ran 104 tests in 53.95 msec all ok
ECC Scalarmult secp224r1 ran 104 tests in 61.60 msec all ok
ECC Scalarmult secp256k1 ran 100 tests in 67.66 msec all ok
ECC Scalarmult secp256r1 ran 104 tests in 80.45 msec all ok
ECC Scalarmult secp384r1 ran 104 tests in 136.24 msec all ok
ECC Scalarmult secp521r1 ran 104 tests in 214.84 msec all ok
ecc_randomized:
ECC randomized brainpool160r1 ran 26 tests in 22.88 msec all ok
ECC randomized brainpool192r1 ran 26 tests in 26.98 msec all ok
ECC randomized brainpool224r1 ran 26 tests in 34.43 msec all ok
ECC randomized brainpool256r1 ran 26 tests in 41.07 msec all ok
ECC randomized brainpool320r1 ran 26 tests in 59.79 msec all ok
ECC randomized brainpool384r1 ran 26 tests in 77.64 msec all ok
ECC randomized brainpool512r1 ran 26 tests in 136.15 msec all ok
ECC randomized frp256v1 ran 26 tests in 40.48 msec all ok
ECC randomized gost_256A ran 26 tests in 40.60 msec all ok
ECC randomized secp160k1 ran 26 tests in 22.02 msec all ok
ECC randomized secp160r1 ran 26 tests in 21.66 msec all ok
ECC randomized secp160r2 ran 26 tests in 21.62 msec all ok
ECC randomized secp192k1 ran 26 tests in 26.04 msec all ok
ECC randomized secp192r1 ran 26 tests in 30.96 msec all ok
ECC randomized secp224k1 ran 26 tests in 34.68 msec all ok
ECC randomized secp224r1 ran 26 tests in 35.92 msec all ok
ECC randomized secp256k1 ran 26 tests in 39.29 msec all ok
ECC randomized secp256r1 ran 26 tests in 47.42 msec all ok
ECC randomized secp384r1 ran 26 tests in 78.55 msec all ok
ECC randomized secp521r1 ran 26 tests in 121.72 msec all ok
ECC randomized sm2p256v1 ran 26 tests in 41.84 msec all ok
ECC randomized x962_p192v2 ran 26 tests in 31.50 msec all ok
ECC randomized x962_p192v3 ran 26 tests in 32.09 msec all ok
ECC randomized x962_p239v1 ran 26 tests in 37.20 msec all ok
ECC randomized x962_p239v2 ran 26 tests in 39.08 msec all ok
ECC randomized x962_p239v3 ran 26 tests in 37.74 msec all ok
ecc_unit:
ECC Unit ran 125 tests all ok
ecdsa_invalid:
ECDSA invalid keys ran 78 tests in 490.47 msec all ok
ecdsa_keygen:
ECDSA frp256v1 keygen ran 15 tests in 36.25 msec all ok
ECDSA secp256r1 keygen ran 15 tests in 42.19 msec all ok
ECDSA secp384r1 keygen ran 15 tests in 72.63 msec all ok
ECDSA secp521r1 keygen ran 15 tests in 123.84 msec all ok
ecdsa_sign:
ECDSA/EMSA1(SHA-1) signature generation ran 8 tests in 26.51 msec all ok
ECDSA/EMSA1(SHA-224) signature generation ran 368 tests in 939.30 msec all ok
ECDSA/EMSA1(SHA-256) signature generation ran 360 tests in 889.53 msec all ok
ECDSA/EMSA1(SHA-384) signature generation ran 360 tests in 897.37 msec all ok
ECDSA/EMSA1(SHA-512) signature generation ran 360 tests in 891.00 msec all ok
ecdsa_unit:
ECDSA Unit ran 49 tests all ok
entropy:
hostname:
Hostname ran 21 tests in 0.07 msec all ok
iv_carryover:
CBC IV carry ran 4 tests all ok
mp_unit:
bigint_cnd_abs ran 6 tests all ok
bigint_cnd_add ran 4 tests all ok
bigint_cnd_sub ran 4 tests all ok
bigint_cnd_swap ran 12 tests all ok
nist_redc:
NIST P-192 reduction ran 34 tests in 3.68 msec all ok
NIST P-224 reduction ran 34 tests in 4.53 msec all ok
NIST P-256 reduction ran 34 tests in 5.97 msec all ok
NIST P-384 reduction ran 34 tests in 10.72 msec all ok
NIST P-521 reduction ran 34 tests in 17.32 msec all ok
ocsp:
OCSP request check ran 3 tests all ok
OCSP request encoding ran 3 tests all ok
OCSP response certificate access ran 3 tests all ok
OCSP response parsing ran 3 tests all ok
octetstring:
OctetString ran 14 tests all ok
os_utils:
OS::get_high_resolution_clock ran 2 tests all ok
OS::get_process_id ran 2 tests all ok
OS::get_processor_timestamp ran 1 tests all ok
OS::get_system_timestamp_ns ran 2 tests all ok
pem:
PEM encoding ran 5 tests all ok
pk_pad_eme:
pk_pad_emsa_unit:
EMSA_name_tests ran 10 tests all ok
pk_workfactor:
DL_Exponent_Size work factor calculation ran 6 tests in 0.02 msec all ok
RSA_Strength work factor calculation ran 8 tests in 0.04 msec all ok
pkcs_hash_id:
PKCS hash id for MD5 ran 2 tests all ok
PKCS hash id for RIPEMD-160 ran 2 tests all ok
PKCS hash id for SHA-160 ran 2 tests all ok
PKCS hash id for SHA-224 ran 2 tests all ok
PKCS hash id for SHA-256 ran 2 tests all ok
PKCS hash id for SHA-3(224) ran 2 tests all ok
PKCS hash id for SHA-3(256) ran 2 tests all ok
PKCS hash id for SHA-3(384) ran 2 tests all ok
PKCS hash id for SHA-3(512) ran 2 tests all ok
PKCS hash id for SHA-384 ran 2 tests all ok
PKCS hash id for SHA-512 ran 2 tests all ok
PKCS hash id for SHA-512-256 ran 2 tests all ok
PKCS hash id for SM3 ran 2 tests all ok
PKCS hash id for Tiger(24,3) ran 2 tests all ok
rsa_blinding:
rsa_encrypt:
rsa_kem:
rsa_keygen:
RSA 1024 keygen ran 15 tests in 212.91 msec all ok
RSA 1280 keygen ran 15 tests in 1.10 sec all ok
rsa_pss:
RSA/PSSR(SHA-224,MGF1,15) signature generation ran 80 tests in 76.22 msec all ok
RSA/PSSR(SHA-224,MGF1,28) signature generation ran 80 tests in 201.01 msec all ok
RSA/PSSR(SHA-256,MGF1,20) signature generation ran 80 tests in 74.81 msec all ok
RSA/PSSR(SHA-256,MGF1,32) signature generation ran 80 tests in 201.07 msec all ok
RSA/PSSR(SHA-384,MGF1,25) signature generation ran 80 tests in 74.92 msec all ok
RSA/PSSR(SHA-384,MGF1,48) signature generation ran 80 tests in 194.61 msec all ok
RSA/PSSR(SHA-512,MGF1,30) signature generation ran 80 tests in 75.03 msec all ok
RSA/PSSR(SHA-512,MGF1,62) signature generation ran 80 tests in 194.61 msec all ok
rsa_pss_raw:
RSA/PSSR_Raw(SHA-224,MGF1,15) signature generation ran 80 tests in 75.39 msec all ok
RSA/PSSR_Raw(SHA-224,MGF1,28) signature generation ran 80 tests in 193.23 msec all ok
RSA/PSSR_Raw(SHA-256,MGF1,20) signature generation ran 80 tests in 74.52 msec all ok
RSA/PSSR_Raw(SHA-256,MGF1,32) signature generation ran 80 tests in 193.26 msec all ok
RSA/PSSR_Raw(SHA-384,MGF1,25) signature generation ran 80 tests in 74.46 msec all ok
RSA/PSSR_Raw(SHA-384,MGF1,48) signature generation ran 80 tests in 193.32 msec all ok
RSA/PSSR_Raw(SHA-512,MGF1,30) signature generation ran 80 tests in 74.66 msec all ok
RSA/PSSR_Raw(SHA-512,MGF1,62) signature generation ran 80 tests in 194.63 msec all ok
rsa_sign:
-> CRASH

However, building with configure.py --cc=msvc --os=windows , results in botan-test.exe being able to complete its battery of tests and output: Tests complete ran 2594122 tests in 99.16 sec 1 tests failed.

Should I be concerned?

randombit commented 6 years ago

A downside of the build being very configurable is that it's basically impossible to test every possible configuration. There is a script that tests the result of disabling any single module, but testing all permutations is not really possible and so sometimes bugs are missed.

In this case, the problem was that if emsa_pkcs1 was enabled, but a hash function that was being used for one of the tests was disabled (in this case, MD5) a crash would ensue. Should be fixed now, thanks for mentioning

DevSidious commented 6 years ago

botan-testsuccessfully completes its tests now. Tested on both Windows and Linux, with

configure.py --cc=msvc --os=windows --minimized-build --enable-modules=x509,ecdsa,rsa,emsa_pkcs1,sha2_64,aes,aes_ni,aes_ssse3

and

./configure.py --cpu=x86_64 --os=linux --minimized-build --enable-modules=x509,ecdsa,rsa,emsa_pkcs1,sha2_64,aes,aes_ni,aes_ssse3

On Windows, botan-test.exe completes with Tests complete ran 83582 tests in 21.58 sec 32 tests failed while on Linux botan-test outputs Tests complete ran 83584 tests in 17.62 sec 32 tests failed. 83582 tests on Windows versus 83584 tests on Linux. Is this normal?

randombit commented 6 years ago

There are a few time related tests (checking handling of dates post-2038) that are skipped if time_t is 32-bit, so if your Windows build is 32-bit that might be why.

neusdan commented 6 years ago

Did you see that 32 tests failed?

randombit commented 6 years ago

I did. That's the recently added BSI tests, a couple are checking that an MD5-signed cert doesn't verify. Which it doesn't except now instead of validation failing because it is an untrusted hash it fails because its an unknown hash.

Wasn't sure of the right fix, maybe just disable the first couple of tests if MD5 is not in the bulid? Alternately, accept either error as ok.

DevSidious commented 6 years ago

Both builds are 64-Bit. I will investigate further and open a new thread, because I don't want to confuse things and diverge too far from the subject.

Thank you for the help. You can close this if you wish.

randombit commented 6 years ago

Indeed we're pretty far off the original topic now.