aws / s2n-tls

An implementation of the TLS/SSL protocols
https://aws.github.io/s2n-tls/usage-guide/
Apache License 2.0
4.51k stars 704 forks source link

s2n_aead_aes_test times out occasionally #4472

Open maddeleine opened 5 months ago

maddeleine commented 5 months ago

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

We have noticed the s2n_aead_aes_test will occasionally take way longer than usual to run and then timeout. This appears to be flaky as a retry fixes the problem. Here's an example of the failing test.

Solution:

Figure out what is causing the flakiness. It's possible it has something to do with not having aes-optimized hardware available, but that's just a theory. Investigation is needed.

goatgoose commented 5 months ago

After investigating this issue a bit, it seems that this isn't a problem with the aead_aes test itself, but rather an issue with the kTLS CI job running all tests very slowly sometimes. The slow CI job causes the aead_aes test to timeout since it's the slowest unit test.

I compared a successful kTLS test run to a kTLS test run with the aead_aes test timing out, and many of the other tests were also slow:

Test Success Duration (s) Timeout Duration (s) % Increase
s2n_quic_transport_params_extension_test 0.05 521.34 1042580%
s2n_ssl_prf_test 0.04 83.91 209675%
s2n_tls13_secrets_rfc8448_test 0.07 36.38 51871%
s2n_ktls_io_sendfile_test 0.08 41.32 51550%
s2n_pq_kem_test 0.05 23.2 46300%
s2n_record_size_test 0.09 23.39 25889%

We should determine why the kTLS job is sometimes running some of the tests very slowly. It might be related to how qemu is configured.