AES-NI ninja failure on cross-compiling - Ubuntu 18.04 - VirtualBox

[pqfan]

Greetings,

Initially I want to thank all the involved contributors for this great project.

I use liboqs along with openssl-oqs and Microsoft's PQCrypto-VPN project, and always want to have the latest update when a new release comes out.

I should state that I am not such experience in coding, so my apologies for any possible wrong statement.

I encounter a cross-compiling error when I attempt to build at Ubuntu 18.04 on VirtualBox with ninja.

I insert inline the error output below:

I am looking forward hearing from you. Keep up with this great work. Best regards

[448/455] Linking C executable tests/speed_sig.exe
FAILED: tests/speed_sig.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/speed_sig.dir/speed_sig.c.obj  -o tests/speed_sig.exe -Wl,--out-implib,tests/libspeed_sig.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
[449/455] Linking C executable tests/test_kem.exe
FAILED: tests/test_kem.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/test_kem.dir/test_kem.c.obj  -o tests/test_kem.exe -Wl,--out-implib,tests/libtest_kem.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -pthread -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
[450/455] Linking C executable tests/kat_sig.exe
FAILED: tests/kat_sig.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/kat_sig.dir/kat_sig.c.obj  -o tests/kat_sig.exe -Wl,--out-implib,tests/libkat_sig.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
[451/455] Linking C executable tests/example_kem.exe
FAILED: tests/example_kem.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/example_kem.dir/example_kem.c.obj  -o tests/example_kem.exe -Wl,--out-implib,tests/libexample_kem.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
[452/455] Linking C executable tests/kat_kem.exe
FAILED: tests/kat_kem.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/kat_kem.dir/kat_kem.c.obj  -o tests/kat_kem.exe -Wl,--out-implib,tests/libkat_kem.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
[453/455] Linking C executable tests/test_sig.exe
FAILED: tests/test_sig.exe 
: && /usr/bin/x86_64-w64-mingw32-gcc   -Wl,--stack,1677216 tests/CMakeFiles/test_sig.dir/test_sig.c.obj  -o tests/test_sig.exe -Wl,--out-implib,tests/libtest_sig.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/liboqs.a -pthread -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes128_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes128_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes128_dec_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_enc_sch_ni+0x42): undefined reference to `oqs_aes256_enc_sch_block_ni'
lib/liboqs.a(aes.c.obj):aes.c:(.text$oqs_aes256_ecb_dec_sch_ni+0x42): undefined reference to `oqs_aes256_dec_sch_block_ni'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

[dstebila]

Can you post what string you used with cmake?

[pqfan]

Greetings dstebila,

I tried both the original command listed at the README.md "cmake -GNinja -DCMAKE_TOOLCHAIN_FILE=../.CMake/toolchain_windows-amd64.cmake -DOQS_USE_CPU_EXTENSIONS=OFF .."

but also you can assume that I have used the openvpn-build Microsoft script which used the following command:

cmake -GNinja -DCMAKE_TOOLCHAIN_FILE=../.CMake/toolchain_windows-amd64.cmake \ -DCMAKE_INSTALL_PREFIX=${BUILDROOT}/openssl-${OPENSSL_VERSION}/oqs \ -DOQS_USE_CPU_EXTENSIONS=OFF .. || die "cmake"

[dstebila]

Can you try commit https://github.com/open-quantum-safe/liboqs/commit/b8ebeac3f721e443a1e6487c08740c435b09d590?

[pqfan]

Turns out to this

[37/942] Building C object src/common/CMakeFiles/common.dir/aes/aes.c.obj FAILED: src/common/CMakeFiles/common.dir/aes/aes.c.obj /usr/bin/x86_64-w64-mingw32-gcc -Iinclude -Wl,--dynamicbase,--nxcompat -fvisibility=hidden -Werror -Wall -Wextra -Wpedantic -Wstrict-prototypes -Wshadow -Wformat=2 -Wfloat-equal -Wwrite-strings -O3 -fomit-frame-pointer -fdata-sections -ffunction-sections -Wl,--gc-sections -Wno-maybe-uninitialized -Wbad-function-cast -std=gnu11 -MD -MT src/common/CMakeFiles/common.dir/aes/aes.c.obj -MF src/common/CMakeFiles/common.dir/aes/aes.c.obj.d -o src/common/CMakeFiles/common.dir/aes/aes.c.obj -c ../src/common/aes/aes.c ../src/common/aes/aes.c: In function ‘OQS_AES128_ECB_enc_sch’: ../src/common/aes/aes.c:78:1: error: embedding a directive within macro arguments is not portable [-Werror]

if defined(OQS_USE_CPU_EXTENSIONS)

^ ../src/common/aes/aes.c:85:1: error: embedding a directive within macro arguments is not portable [-Werror]

endif

^ ../src/common/aes/aes.c:93:1: error: embedding a directive within macro arguments is not portable [-Werror]

if defined(OQS_USE_CPU_EXTENSIONS)

^ ../src/common/aes/aes.c:100:1: error: embedding a directive within macro arguments is not portable [-Werror]

endif

^ ../src/common/aes/aes.c:108:1: error: embedding a directive within macro arguments is not portable [-Werror]

if defined(OQS_USE_CPU_EXTENSIONS)

^ ../src/common/aes/aes.c:115:1: error: embedding a directive within macro arguments is not portable [-Werror]

endif

^ ../src/common/aes/aes.c:134:1: error: embedding a directive within macro arguments is not portable [-Werror]

define BE_TO_UINT32(n) (uint32_t)((((uint8_t ) &(n))[0] << 24) | (((uint8_t ) &(n))[1] << 16) | (((uint8_t ) &(n))[2] << 8) | (((uint8_t ) &(n))[3] << 0))

^ ../src/common/aes/aes.c:169:0: error: unterminated argument list invoking macro "C_OR_NI" }

../src/common/aes/aes.c:73:2: error: ‘C_OR_NI’ undeclared (first use in this function) C_OR_NI( ^~~ ../src/common/aes/aes.c:73:2: note: each undeclared identifier is reported only once for each function it appears in ../src/common/aes/aes.c:73:2: error: expected ‘;’ at end of input ../src/common/aes/aes.c:73:2: error: expected declaration or statement at end of input ../src/common/aes/aes.c:72:44: error: unused parameter ‘plaintext’ [-Werror=unused-parameter] void OQS_AES128_ECB_enc_sch(const uint8_t plaintext, const size_t plaintext_len, const void schedule, uint8_t ciphertext) { ^~~~~ ../src/common/aes/aes.c:72:68: error: unused parameter ‘plaintext_len’ [-Werror=unused-parameter] void OQS_AES128_ECB_enc_sch(const uint8_t plaintext, const size_t plaintext_len, const void schedule, uint8_t ciphertext) { ^~~~~ ../src/common/aes/aes.c:72:95: error: unused parameter ‘schedule’ [-Werror=unused-parameter] void OQS_AES128_ECB_enc_sch(const uint8_t plaintext, const size_t plaintext_len, const void schedule, uint8_t ciphertext) { ^~~~ ../src/common/aes/aes.c:72:114: error: unused parameter ‘ciphertext’ [-Werror=unused-parameter] void OQS_AES128_ECB_enc_sch(const uint8_t plaintext, const size_t plaintext_len, const void schedule, uint8_t ciphertext) { ^~~~~~ cc1: all warnings being treated as errors [42/942] Building C object src/kem/sike/CMakeFiles/sike.dir/external/P610/P610_compressed.c.obj ninja: build stopped: subcommand failed. FATAL: ninja FATAL: build x86_64 >&2 Traceback (most recent call last): File "build.py", line 259, in build_openvpn_windows() File "build.py", line 217, in build_openvpn_windows run_command(['./windows-nsis/build-complete']) File "build.py", line 43, in run_command raise RuntimeError('Command failed') RuntimeError: Command failed

[dstebila]

It's hard for me to debug since I don't have a similar build system. I've pushed a potential fix, can you try the latest commit on branch ds-aes-ni-def? Also, what's the version of gcc that mingw is using?

[pqfan]

Going to try it right away

gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

[pqfan]

Ok! Seems to work just fine now!

For your records the system uses the GNU 7.3.0 Compiler, if I understood your question well.

[pqfan]

I confirm I successfully cross-compiled and built a custom PQCrypto-VPN with Microsoft's build system and it works nicely with openssl-oqs latest release.

Thank you a lot for your work on this project and your support!

[Hatzen]

I m facing the exact same issue with Release 0.4.0, but #815 fixed it for me as well. I think the fix should be included in the current version.