using dynamic library built with `CRYPTOPP_DISABLE_ASM` can cause SEGV.

The following code ends with Segmentation fault, at least on x86/Freebsd/v8.7.0 and x64/Arch/c364032d4c(current master).

#!/usr/bin/env bash
set -euxo pipefail

dir=$PWD/classlayout
mkdir -p $dir
make clean
make CRYPTOPP_CPPFLAGS='-DNDEBUG -DCRYPTOPP_DISABLE_ASM' PREFIX=$dir dynamic libcryptopp.pc
make CRYPTOPP_CPPFLAGS='-DNDEBUG -DCRYPTOPP_DISABLE_ASM' PREFIX=$dir install-lib

cat > $dir/main.cc <<EOF
#include <cryptopp/osrng.h>
#include <cryptopp/sha.h>
int main(){
    CryptoPP::AutoSeededRandomPool asrp;
    CryptoPP::SHA256 sha256;
    return 0;
}
EOF

c++ -o $dir/main $dir/main.cc $(PKG_CONFIG_LIBDIR=$dir/lib/pkgconfig pkgconf --cflags --libs libcryptopp)
LD_LIBRARY_PATH=$dir/lib $dir/main

In the dynamic library, define CRYPTOPP_DISABLE_ASM will also make CRYPTOPP_BOOL_ALIGN16 defined to 0. In the app main, CRYPTOPP_DISABLE_ASM is not defined, CRYPTOPP_BOOL_ALIGN16 is 1 on x86/x64. The different CRYPTOPP_BOOL_ALIGN16 value will result in different class size and layout of FixedSizeAllocatorWithCleanup<T, S, A, true>, which is nested in SHA256. However, the template class FixedSizeAllocatorWithCleanup is also instantiated in app main exported(I reckon crash only happens to PIE code, non-PIE will not export symbols automatically.). The whole binding process of dynamic symbols is fairly complicated, unfortunately I don't fully understand it. The observed behavior is some symbols in libcryptopp.so are binding to main, so two different class layouts are used by libcryptopp.so, and crash ensued.

There are three probable solutions:

Add CRYPTOPP_DISABLE_ASM to cflags in libcryptopp.pc, if defined.
Make class FixedSizeAllocatorWithCleanup layout independent to CRYPTOPP_DISABLE_ASM.
Mark template classes as invisible to other DSOs in installed header files.

The first one seems the simplest and most straightforward.

weidai11 / cryptopp

using dynamic library built with `CRYPTOPP_DISABLE_ASM` can cause SEGV. #1191