./botan-test crashes when statically linked

[webmaster128]

When I build the current master (https://github.com/randombit/botan/commit/1bf1490726d859596ac95c78c9a7763b8d420b2d), botan-test crashes

$ ./botan-test
[1]    520 segmentation fault (core dumped)  ./botan-test

while the help output works

$ ./botan-test --help
Usage: ./botan-test [suite]
Suites: all (default), block, hash, bigint, rsa, ecdsa, ...

as well as running the tests individually

$ suites=(block modes aead ocb stream hash mac pbkdf kdf keywrap transform rngs passhash9 bcrypt cryptobox tss rfc6979 bigint rsa rw dsa nr dh dlies elgamal ecc_pointmul ecdsa gost_3410 curve25519 mceliece ecc_unit ecdsa_unit ecdh_unit pk_keygen cvc x509 nist_x509 tls); \
for suite in $suites; do echo "$suite"; ./botan-test "$suite"; done

Here some tests fail for different reasons but the app does not crash.

The same goes for Linux (g++), OS X (clang) and Windows (MSVC2013).

Can anyone reproduce this behaviour?

[randombit]

A clean checkout is working fine for me on Linux g++ 4.9.1

The process crashes immediately after starting, before producing output? Can you get a backtrace?

[webmaster128]

Jap it crashes directly before any output. How do I produces a backtrace (on one platform of you choice)?

[webmaster128]

Here you go (OS X): https://gist.github.com/webmaster128/c74a7e561c5a01c5ef0c

Running the ocb test alone reproduces the crash:

$ ./botan-test ocb
Segmentation fault: 11

[randombit]

From the dylib listing it looks like botan-test was statically linked. That is almost certainly the problem. I'll need to fix the tests but in the short term the workaround is to link to libbotan.so. See #52 for details, but basically the current registry system doesn't work well with static linking.

[webmaster128]

Jap, it is statically linked.

$ cd botan
$ python configure.py --disable-shared --with-build-dir=../build-botan --link-method=hardlink
$ make --jobs 2 -f ../build-botan/Makefile

In general it should work like that, right?

[mouse07410]

Yes, that is my understanding.

Sent from my iPad

On Mar 13, 2015, at 11:16, Simon Warta notifications@github.com<mailto:notifications@github.com> wrote:

Jap, it is statically linked.

cd botan python configure.py --disable-shared --with-build-dir=../build-botan --link-method=hardlink make --jobs 2 -f ../build-botan/Makefile

In general it should work like that, right?

— Reply to this email directly or view it on GitHubhttps://github.com/randombit/botan/issues/58#issuecomment-79034813.

[webmaster128]

I updated my testing command since the one I used above was buggy. See ./botan-tester.sh

./botan-tester.sh 
Run test 'block' ...
Tests all ok
Run test 'modes' ...
Tests all ok
Run test 'aead' ...
Tests all ok
Run test 'ocb' ...
./botan-tester.sh: Zeile 5: 18110 Speicherzugriffsfehler  (Speicherabzug geschrieben) ./botan-test "$suite"
Error in last test! Return value: 139

It turns out that the ocb test cashes, no matter if the lib is build statically or dynamically. I did:

$ cd ../botan
$ python configure.py  --with-build-dir=../build-botan --link-method=hardlink 
$ make --jobs 2 -f ../build-botan/Makefile

./botan-test is always statically linked, right? At least ./botan-test block runs without any libbotan-1.11.so available locally.

[randombit]

This output:

Run test 'block' ... Tests all ok

is pretty ominous - it suggests nothing is running, probably because boost fs was not enabled and write now fs.cpp has not been specialized for Windows. So more things may be broken than OCB.

[webmaster128]

What is the expected output of ./botan-test block? I get the same Tests all ok on Windows as well.

[randombit]

Something like:

AES-128 385 tests all ok
AES-192 449 tests all ok
AES-256 513 tests all ok
Blowfish 60 tests all ok
Camellia-128 6 tests all ok
Camellia-192 3 tests all ok
Camellia-256 5 tests all ok
Cascade(Serpent,Twofish) 3 tests all ok
Cascade(Serpent,AES-256) 2 tests all ok
Cascade(Serpent,CAST-128) 1 tests all ok
CAST-128 41 tests all ok
CAST-256 13 tests all ok
DES 322 tests all ok
DESX 9 tests all ok
TripleDES 58 tests all ok
GOST-28147-89(R3411_94_TestParam) 18 tests all ok
GOST-28147-89(R3411_CryptoPro) 10 tests all ok
IDEA 541 tests all ok
KASUMI 3 tests all ok
Lion(SHA-1,ARC4,64) 1 tests all ok
MARS 1084 tests all ok
MISTY1 31 tests all ok
Noekeon 4 tests all ok
RC2 29 tests all ok
RC5(12) 46 tests all ok
RC5(16) 41 tests all ok
RC6 1219 tests all ok
SAFER-SK(10) 12 tests all ok
SEED 4 tests all ok
Serpent 1044 tests all ok
TEA 38 tests all ok
Threefish-512 4 tests all ok
Twofish 1107 tests all ok
XTEA 36 tests all ok
Tests all ok

The no outputs would be consistent with list_all_readable_files_in_or_under in fs.cpp returning the empty vector instead of a list of files to read.

[randombit]

The tests all OK line should be printing the # of tests that ran, which would have made this issue more obvious probably.

[webmaster128]

Does the test look for files at runtime? Can that be caused by building botan into a different directory than default?

[webmaster128]

Okay, everything depends on the working directory.

cd /workspace/build-botan
./botan-test block

just outputs Tests all ok whereas

cd /workspace/botan
../build-botan/botan-test block

shows a long list beginning with

AES-128 test 1 failed: Unknown block cipher AES-128
AES-128 test 2 failed: Unknown block cipher AES-128
AES-128 test 3 failed: Unknown block cipher AES-128
AES-128 test 4 failed: Unknown block cipher AES-128
AES-128 test 5 failed: Unknown block cipher AES-128
AES-128 test 6 failed: Unknown block cipher AES-128
AES-128 test 7 failed: Unknown block cipher AES-128
AES-128 test 8 failed: Unknown block cipher AES-128
AES-128 test 9 failed: Unknown block cipher AES-128
AES-128 test 10 failed: Unknown block cipher AES-128
AES-128 test 11 failed: Unknown block cipher AES-128
AES-128 test 12 failed: Unknown block cipher AES-128
AES-128 test 13 failed: Unknown block cipher AES-128

which is no surprise when run_tests_in_dir(TEST_DATA_DIR "block", test_bc); with TEST_DATA_DIR "src/tests/data/" is called.

[randombit]

OK, now this makes sense. OCB crashes because get_block_cipher("AES-128") returns null and the test doesn't expect that (I'll fix this). This, and the errors you got, would be consistent with a build that was statically linked but not built with --via-amalgamation, so the registry system is not picking up any of the ciphers.

[webmaster128]

Cool, thanks. Is static linking without amalgamation not allowed? Or is it just buggy? (I am not an expert in these linking questions)

[randombit]

I'm honestly not sure yet (re static linking without the amalgamation being unsupported or supported-but-broken) but definitely the situation above (silently broken build) is bad. However, regardless of supported or not in the long run, right now (since 1.11.14) static linking doesn't work unless done with --via-amalgamation.

[randombit]

Oh, and re: working directory, thanks for pointing that out, I'll find some way to direct the tests to the right dir for builds placed outside the source directory.

[webmaster128]

I'll find some way to direct the tests to the right dir for builds placed outside the source directory.

One could argue that this is the user's problem. makeing is also only possible from the source dir. In my opinion in a first step it is important to add additional checks such as file existence and tell the user what Botan was looking for.

[webmaster128]

Most of that one is done. I can now run tests for out-of-tree-builds when the botan tree is the working directory. I'm going to document it at some point in time. The last open question in this thread is now in #128.