FreeBSD + qemu seeing slow aead_aes test

dougch commented 11 months ago

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

While looking into FreeBSD CI issues, discovered that on qemu the s2n_aead_aes_test takes 350+ seconds, which is longer than the entire test run on native x86. This might be related to qemu, but this is worth looking into under virtualbox as well (what the GitHub action uses) and with KVM support.

On FreeBSD 13.1

./.github/s2n_bsd.sh
-- The C compiler identification is Clang 13.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detected CMAKE_SYSTEM_PROCESSOR as amd64
-- Detected 64-Bit system
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found crypto: /usr/lib/libcrypto.a
-- LibCrypto Include Dir: /usr/include
-- LibCrypto Shared Lib:  /usr/lib/libcrypto.so
-- LibCrypto Static Lib:  /usr/lib/libcrypto.a
-- Using libcrypto from the cmake path
-- CMAKE_AR found: /usr/bin/llvm-ar
-- CMAKE_RANLIB found: /usr/bin/llvm-ranlib
-- CMAKE_OBJCOPY found: /usr/bin/llvm-objcopy
-- feature S2N_ATOMIC_SUPPORTED: TRUE
-- feature S2N_CLOEXEC_SUPPORTED: TRUE
-- feature S2N_CLOEXEC_XOPEN_SUPPORTED: TRUE
-- feature S2N_CLONE_SUPPORTED: FALSE
-- feature S2N_CPUID_AVAILABLE: TRUE
-- feature S2N_DIAGNOSTICS_POP_SUPPORTED: TRUE
-- feature S2N_DIAGNOSTICS_PUSH_SUPPORTED: TRUE
-- feature S2N_EXECINFO_AVAILABLE: TRUE
-- feature S2N_FALL_THROUGH_SUPPORTED: TRUE
-- feature S2N_FEATURES_AVAILABLE: FALSE
-- feature S2N_KTLS_SUPPORTED: FALSE
-- feature S2N_KYBER512R3_AVX2_BMI2_SUPPORTED: TRUE
-- feature S2N_KYBER512R3_M256_INTRINSICS_SUPPORTED: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_MD5_SHA1_HASH: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_MD_CTX_SET_PKEY_CTX: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_EVP_RC4: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_FLAG_NO_CHECK_TIME: TRUE
-- feature S2N_LIBCRYPTO_SUPPORTS_HKDF: FALSE
-- feature S2N_LIBCRYPTO_SUPPORTS_KYBER: FALSE
-- feature S2N_LINUX_SENDFILE: FALSE
-- feature S2N_MADVISE_SUPPORTED: TRUE
-- feature S2N_MINHERIT_SUPPORTED: TRUE
-- feature S2N___RESTRICT__SUPPORTED: TRUE
-- feature S2N_STACKTRACE: TRUE
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /usr/bin/cc
-- feature S2N_KYBER512R3_AVX2_BMI2: TRUE
-- Running tests with environment: S2N_DONT_MLOCK=1
-- Configuring done (16.6s)
-- Generating done (1.4s)
-- Build files have been written to: /root/s2n-tls/build
...snip...
245/251 Test #248: s2n_wildcard_hostname_test .......................   Passed    0.03 sec
        Start 249: s2n_x509_validator_certificate_signatures_test
246/251 Test #245: s2n_tls_prf_test .................................   Passed    0.13 sec
        Start 250: s2n_x509_validator_test
247/251 Test #249: s2n_x509_validator_certificate_signatures_test ...   Passed    0.06 sec
        Start 251: s2n_x509_validator_time_verification_test
248/251 Test #250: s2n_x509_validator_test ..........................   Passed    0.39 sec
249/251 Test #251: s2n_x509_validator_time_verification_test ........   Passed    0.47 sec
250/251 Test #239: s2n_tls13_secrets_test ...........................   Passed   18.01 sec
251/251 Test   #2: s2n_aead_aes_test ................................   Passed  351.32 sec

100% tests passed, 0 tests failed out of 251

Solution:

A description of the possible solution in terms of S2N architecture. Highlight and explain any potentially controversial design decisions taken.

Unclear until more investigation is done.

Requirements / Acceptance Criteria:

What must a solution address in order to solve the problem? How do we know the solution is complete?

RFC links: Links to relevant RFC(s)
Related Issues: Link any relevant issues
Will the Usage Guide or other documentation need to be updated?
Testing: How will this change be tested? Call out new integration tests, functional tests, or particularly interesting/important unit tests.
- Will this change trigger SAW changes? Changes to the state machine, the s2n_handshake_io code that controls state transitions, the DRBG, or the corking/uncorking logic could trigger SAW failures.
- Should this change be fuzz tested? Will it handle untrusted input? Create a separate issue to track the fuzzing work.

Out of scope:

Is there anything the solution will intentionally NOT address?

dougch commented 11 months ago

Confirmed that the kTLS test on qemu also suffers from slowness for aead_aes at almost 800 seconds:

2: Running /home/codebuild/s2n-tls/tests/unit/s2n_aead_aes_test.c ... PASSED   32098261 tests
251/251 Test   #2: s2n_aead_aes_test ................................   Passed  759.61 sec

jmayclin commented 11 months ago

I wonder if this is related to hardware acceleration issues? The AEAD test is normally quite slow, so perhaps if it's using a software implementation that might explain the extra time.

https://github.com/cross-platform-actions/action/issues/47 https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/

aws / s2n-tls