search

Mbed-TLS / mbedtls

An open source, portable, easy to use, readable and flexible TLS library, and reference implementation of the PSA Cryptography API. Releases are on a varying cadence, typically around 3 - 6 months between releases.
https://www.trustedfirmware.org/projects/mbed-tls/
Other
5.23k stars 2.56k forks source link

Possible race condition in ssl-opt.sh #2670

Closed mpg closed 5 years ago

mpg commented 5 years ago

Description

Bug

Expected behavior

In the CI, ssl-opt.sh passes reliably except when it catches an actual issue in the code being tested.

Actual behavior

It sometimes fails with errors such as:

DTLS fragmenting: 3d, gnutls client, DTLS 1.0 .......................... FAIL

  ! server or client failed to reach handshake stage

  ! outputs saved to o-XXX-576.log

  ! server output:

../programs/ssl/ssl_server2 server_addr=127.0.0.1 server_port=13621 allow_sha1=1 dtls=1 debug_level=2              crt_file=data_files/server7_int-ca.crt              key_file=data_files/server7.key              hs_timeout=250-60000 mtu=512 force_version=dtls1

  . Seeding the random number generator... ok

  . Loading the CA root certificate ... ok (0 skipped)

  . Loading the server cert. and key... ok

  . Bind on udp://127.0.0.1:13621/ ... ok

  . Setting up the SSL/TLS structure... ok

  . Waiting for a remote connection ... interrupted by SIGTERM (in net_accept())

  . Cleaning up...ssl_tls.c:10142: |2| => free

ssl_tls.c:10207: |2| <= free

 done.

  ! ========================================================

  ! client output:

echo 'GET / HTTP/1.0' | /usr/local/gnutls-3.6.5/bin/gnutls-cli --x509cafile data_files/test-ca_cat12.crt -p 23621 -u --insecure 127.0.0.1

*** Fatal error: Error in the push function.

Could not connect to 127.0.0.1:23621: Connection refused

Processed 2 CA certificate(s).

Resolving '127.0.0.1:23621'...

Connecting to '127.0.0.1:23621'...

EXIT: 1

  ! ========================================================

  ! proxy output:

../programs/test/udp_proxy server_addr=127.0.0.1 server_port=13621 listen_addr=127.0.0.1 listen_port=23621 seed=1 drop=8 delay=8 duplicate=8

  . Connect to server on UDP/127.0.0.1/13621 ... ok

  . Bind on UDP/127.0.0.1/23621 ... ok

  . Waiting for a remote connection ...

Steps to reproduce

I believe this to be a race condition, so hard to reproduce. Here's one instance of it happening: https://jenkins-internal.mbed.com/blue/organizations/jenkins/mbed-tls-restricted-pr/detail/PR-567-head/13/pipeline/

Analysis

It looks like the client is trying to connect to the proxy before the proxy had time to start listening. We have logic to check that the server is listening, but none to check the the proxy is also listening before we start the client.

ciarmcom commented 5 years ago

ARM Internal Ref: IOTSSL-2885

mpg commented 5 years ago

This was fixed by https://github.com/ARMmbed/mbedtls/pull/2815