Compiling botan with llvm/clang to webassembly

[courteous]

I am trying to compile botan with clang/llvm for webassembly with the following command

./configure.py --minimized-build --cc=clang --os=llvm --cpu=llvm 
   INFO: ./configure.py invoked with options "--minimized-build --cc=clang --os=llvm --cpu=llvm"
   INFO: Autodetected platform information: OS="Linux" machine="x86_64" proc="x86_64"
   INFO: Shared libs not supported on llvm, disabling shared lib support
   INFO: Auto-detected compiler version 6.0
   INFO: Auto-detected compiler arch x86_64
WARNING: Configured target is llvm but compiler probe indicates x86_64
   INFO: Target is clang:6.0-llvm-llvm
   INFO: Skipping (incompatible CPU): aes_armv8 aes_ni aes_power8 aes_ssse3 chacha_avx2 chacha_sse2 clmul clmul_ssse3 idea_sse2 pmull rdrand_rng rdseed serpent_avx2 sha1_armv8 sha1_sse2 sha1_x86 sha2_32_armv8 sha2_32_bmi2 sha2_32_x86 shacal2_x86 simd_avx2 sm4_armv8 threefish_512_avx2
   INFO: Skipping (incompatible OS): commoncrypto dev_random dyn_load fd_unix getentropy locking_allocator proc_walk system_rng thread_utils win32_stats
   INFO: Skipping (not requested): adler32 aead aes aont aria asn1 auto_rng base32 base64 bcrypt bigint blake2 block blowfish camellia cascade cast128 cast256 cbc cbc_mac ccm cecpq1 certstor_sql certstor_sqlite3 cfb chacha chacha20poly1305 chacha_rng checksum cmac codec_filt comb4p compression crc24 crc32 cryptobox ctr curve25519 des dh dl_algo dl_group dlies dsa eax ec_group ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 ffi filters fpe_fe1 gcm gmac gost_28147 gost_3410 gost_3411 hash hash_id hkdf hmac hmac_drbg hotp http_util idea iso9796 kasumi kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion mac mce mceies md4 md5 mdx_hash mem_pool mgf1 misty1 mode_pad modes mp newhope nist_keywrap noekeon noekeon_simd numbertheory ocb ofb par_hash passhash9 pbes2 pbkdf pbkdf1 pbkdf2 pem pgp_s2k pk_pad pkcs11 poly1305 poly_dbl prf_tls prf_x942 psk_db pubkey rc4 rdrand rfc3394 rfc6979 rmd160 rsa salsa20 scrypt seed serpent serpent_simd sessions_sql sessions_sqlite3 sha1 sha2_32 sha2_64 sha3 shacal2 shacal2_simd shake shake_cipher simd siphash siv skein sm2 sm3 sm4 socket sp800_108 sp800_56a sp800_56c srp6 stateful_rng stream streebog threefish_512 tiger tls tls_cbc tss twofish uuid whirlpool x509 x919_mac xmss xtea xts
   INFO: Skipping (requires external dependency): bearssl boost bzip2 lzma openssl sqlite3 tpm zlib
   INFO: Loading modules: base cpuid entropy hex rng utils
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Botan 2.9.0 (revision git:f396fc92bab378440741174fdbb9144dccb0f90d) (unreleased undated) build setup is complete

but when i issue emmake make, i see that botan is trying to search for gcc instead of a clang i get the following error

make
clang++  -fstack-protector -pthread -std=c++11 -D_REENTRANT -O3 -emit-llvm -fno-use-cxa-atexit -Wall -Wextra -Wpedantic -Wshadow -Wstrict-aliasing -Wstrict-overflow=5 -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual  -Ibuild/include -c src/lib/base/scan_name.cpp -o build/obj/lib/base_scan_name.bc
In file included from src/lib/base/scan_name.cpp:8:
In file included from build/include/botan/scan_name.h:11:
In file included from build/include/botan/types.h:15:
/usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/cstddef:50:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~
1 error generated.
Makefile:86: recipe for target 'build/obj/lib/base_scan_name.bc' failed
make: *** [build/obj/lib/base_scan_name.bc] Error 1

i also tried with the switch

--includedir

since i am tried to point to the stddef.h from the emscripten repository but that did not worked as well.

i do not understand what i am doing wrong here and how to tell botan where the header files are.

since the path above i.e. /usr/lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/cstddef:50:10: is pointing to /usr/include/c++/7.3.0

and obviouslly the stddef.h file is not there and simple locate on that file shows me that it is located under:

/usr/include/linux/stddef.h
/usr/lib/gcc/x86_64-linux-gnu/7/include/stddef.h
/usr/lib/llvm-6.0/lib/clang/6.0.0/include/stddef.h
/usr/src/linux-headers-4.15.0-33/include/linux/stddef.h
/usr/src/linux-headers-4.15.0-33/include/uapi/linux/stddef.h
/usr/src/linux-headers-4.15.0-34/include/linux/stddef.h
/usr/src/linux-headers-4.15.0-34/include/uapi/linux/stddef.h

so how do i tell botan to look in one of those locations or in the custom emscripten directory?

[randombit]

I was not familiar with emmake and it looks like how it works is not compatible with our makefile. (Now that I know about it, I will try to address this.)

This works for me

CXX=em++ ./configure.py  --minimized-build --cc=clang --os=llvm --cpu=llvm

modulo some regressions in the build which I just fixed in b074cb4b5 - Emscripten isn't part of our CI builds right now, and I haven't tested this since last year, so things have rotted slightly.

However it appears even with the fixes, there is some problem, as trying to create a Wasm file results in errors like

Intrinsic has incorrect argument type!
void (i8*, i8*, i32, i1)* @llvm.memcpy.p0i8.p0i8.i32
Intrinsic has incorrect argument type!
void (i8*, i8, i64, i1)* @llvm.memset.p0i8.i64
/usr/lib/emscripten-fastcomp/opt: /home/jack/work/botan/botan-test.bc: error: input module is broken!

I'm not sure what the issue with this is. I can't find anything relevant on the emscripten issue tracker either.

I should mention also, if you are looking for a battle tested crypto library for Wasm, botan is (currently at least) not it. Would be very happy to improve this situation but don't want to you spend a lot of time on something that may not be suitable.

[randombit]

https://stackoverflow.com/questions/51429898/llvm-link-error-when-using-memcpy-in-c-code-and-compiling-with-wasm-target hints that -fno-builtin fixes the problem, but no luck.

[courteous]

well with the following the compilation is starting i.e. Botan 2.9.0 (revision git:b074cb4b554d2e932f4a5be48464f4b2716b3c2c)

emconfigure ./configure.py --minimized-build --cc=clang --os=llvm --cpu=llvm 
emmake make

but then it fails during archivating phase with one warrning

src/lib/utils/os_utils.cpp:408:57: warning: unused parameter 'probe_fn' [-Wunused-parameter]
int OS::run_cpu_instruction_probe(std::function<int ()> probe_fn)

......

/home/user/Projects/emscripten/emsdk/emscripten/1.38.12/emar -o libbotan-2.bc build/obj/lib/base_scan_name.bc build/obj/lib/base_sym_algo.bc build/obj/lib/base_symkey.bc build/obj/lib/codec_hex.bc build/obj/lib/entropy_srcs.bc build/obj/lib/rng.bc build/obj/lib/utils_assert.bc build/obj/lib/utils_calendar.bc build/obj/lib/utils_charset.bc build/obj/lib/utils_cpuid.bc build/obj/lib/utils_cpuid_arm.bc build/obj/lib/utils_cpuid_ppc.bc build/obj/lib/utils_cpuid_x86.bc build/obj/lib/utils_data_src.bc build/obj/lib/utils_exceptn.bc build/obj/lib/utils_filesystem.bc build/obj/lib/utils_mem_ops.bc build/obj/lib/utils_os_utils.bc build/obj/lib/utils_parsing.bc build/obj/lib/utils_read_cfg.bc build/obj/lib/utils_read_kv.bc build/obj/lib/utils_timer.bc build/obj/lib/utils_version.bc
/home/user/Projects/emscripten/emsdk/clang/e1.38.12_64bit/llvm-ar: You must specify at least one of the operations.
OVERVIEW: LLVM Archiver (llvm-ar)

  This program archives bitcode files into single libraries

USAGE: llvm-ar [options] [relpos] [count] <archive-file> [members]...

OPTIONS:
  -M                  - 
  -format             - Archive format to create
    =default          -   default
    =gnu              -   gnu
    =darwin           -   darwin
    =bsd              -   bsd
  -help               - Display available options (-help-hidden for more)
  -plugin=<string>    - plugin (ignored for compatibility
  -print-module-scope - When printing IR for print-[before|after]{-all} always print a module IR
  -version            - Display the version of this program
Makefile:79: recipe for target 'libbotan-2.bc' failed
make: *** [libbotan-2.bc] Error 1

[randombit]

emconfigure for reasons I don't understand is picking llvm-ar to link the bitcode, instead of llvm-link. Wouldn't you want llvm-link for this operation? Anyway, the -o flag is assuming llvm-link syntax.

If this is all wrong, you can edit src/build-data/cc/llvm.txt to modify.

I will not be able to look at this any more until end of the week but feel free to drop any notes in here and I will read it when I can.

[courteous]

I was able to create the archive with the following command

emar cr botan2.bc build/obj/lib/base_scan_name.bc build/obj/lib/base_sym_algo.bc build/obj/lib/base_symkey.bc build/obj/lib/codec_hex.bc build/obj/lib/entropy_srcs.bc build/obj/lib/math_bigint_big_code.bc build/obj/lib/math_bigint_big_io.bc build/obj/lib/math_bigint_big_ops2.bc build/obj/lib/math_bigint_big_ops3.bc build/obj/lib/math_bigint_big_rand.bc build/obj/lib/math_bigint.bc build/obj/lib/math_bigint_divide.bc build/obj/lib/math_mp_comba.bc build/obj/lib/math_mp_core.bc build/obj/lib/math_mp_karat.bc build/obj/lib/math_mp_monty.bc build/obj/lib/math_mp_monty_n.bc build/obj/lib/rng.bc build/obj/lib/utils_assert.bc build/obj/lib/utils_calendar.bc build/obj/lib/utils_charset.bc build/obj/lib/utils_cpuid.bc build/obj/lib/utils_cpuid_arm.bc build/obj/lib/utils_cpuid_ppc.bc build/obj/lib/utils_cpuid_x86.bc build/obj/lib/utils_data_src.bc build/obj/lib/utils_exceptn.bc build/obj/lib/utils_filesystem.bc build/obj/lib/utils_mem_ops.bc build/obj/lib/utils_os_utils.bc build/obj/lib/utils_parsing.bc build/obj/lib/utils_read_cfg.bc build/obj/lib/utils_read_kv.bc build/obj/lib/utils_timer.bc build/obj/lib/utils_version.bc

basically instead of " emar-o " i used " emar cr" i..e [c] - do not warn if the library had to be created r[abfiuRsS] - replace or insert file(s) into the archive

then i used the archive in cmake without any problmes

[randombit]

In ea72317 I've added a Emscripten specific OS target which uses emar and also allows using features which are usable within Emscripten but not within generic LLVM bitcode. Switching to emar instead of llvm-link also seems to fix the problems I was seeing with the builtins (not sure why that made a difference but whatever). With this it is possible to build and run the test suite as WebAssembly.

With that addressed I think this issue can be closed. @courteous feel free to reopen this issue or open an new issue if you run into further problems.