search

Mbed-TLS / mbedtls

An open source, portable, easy to use, readable and flexible TLS library, and reference implementation of the PSA Cryptography API. Releases are on a varying cadence, typically around 3 - 6 months between releases.
https://www.trustedfirmware.org/projects/mbed-tls/
Other
5.2k stars 2.55k forks source link

Failure connecting to TLSv1.3 #9415

Open viric opened 1 month ago

viric commented 1 month ago

Summary

I'm using OBS, that uses mbedtls to connect to a rtmps server. My server uses TLS1.3 implemented in nodejs' tls package. If I "curl" to that URL, all works. But obs fails to conect with mbedtls. First, I patched obs for the #8401, because that was the first problem. The second problem I don't know what itis.

System information

Mbed TLS version (number or commit id): 3.6.0 Operating system and version: NixOS Linux 24.05 Configuration (if not default, please attach mbedtls_config.h): default Compiler and options (if you used a pre-built binary, please indicate how you obtained it): gcc, nothing strange Additional environment information:

Expected behavior

OBS connects well to the rtmps server

Actual behavior

It fails to connect

Steps to reproduce

Try to connect to my rtmps server.

Additional information

Here is the end of debug log, removing big amounts of data

/build/source/library/ssl_msg.c:3231: 0x58ae540: handshake message: msglen = 265, type = 4, hslen = 265
/build/source/library/ssl_msg.c:4261: 0x58ae540: <= read record
/build/source/library/ssl_msg.c:5597: 0x58ae540: received post-handshake message
/build/source/library/ssl_msg.c:5585: 0x58ae540: NewSessionTicket received
info: RTMPSockBuf_Fill, recv returned -26880. GetSockError(): 13 (unknown error)
/build/source/library/ssl_msg.c:6187: 0x58ae540: => write close notify
/build/source/library/ssl_msg.c:5168: 0x58ae540: => send alert message
/build/source/library/ssl_msg.c:5169: 0x58ae540: send alert level=1 message=0
/build/source/library/ssl_msg.c:2943: 0x58ae540: => write record
/build/source/library/ssl_msg.c:0935: 0x58ae540: => encrypt buf
/build/source/library/ssl_msg.c:0957: 0x58ae540: dumping 'before encrypt: output payload' (2 bytes)
/build/source/library/ssl_msg.c:0957: 0x58ae540: 0000:  01 00                                            ..
/build/source/library/ssl_msg.c:1003: 0x58ae540: dumping 'CID' (0 bytes)
/build/source/library/ssl_msg.c:1182: 0x58ae540: dumping 'IV used (internal)' (12 bytes)
/build/source/library/ssl_msg.c:1182: 0x58ae540: 0000:  c6 75 5c b0 71 64 13 7a 38 73 54 13              .u\.qd.z8sT.
/build/source/library/ssl_msg.c:1184: 0x58ae540: dumping 'IV used (transmitted)' (0 bytes)
/build/source/library/ssl_msg.c:1187: 0x58ae540: dumping 'additional data used for AEAD' (5 bytes)
/build/source/library/ssl_msg.c:1187: 0x58ae540: 0000:  17 03 03 00 20                                   .... 
/build/source/library/ssl_msg.c:1189: 0x58ae540: before encrypt: msglen = 16, including 0 bytes of padding
/build/source/library/ssl_msg.c:1223: 0x58ae540: dumping 'after encrypt: tag' (16 bytes)
/build/source/library/ssl_msg.c:1223: 0x58ae540: 0000:  32 8e 75 31 74 d1 1a 2a 36 01 b3 28 cc b5 93 f9  2.u1t..*6..(....
/build/source/library/ssl_msg.c:1474: 0x58ae540: <= encrypt buf
/build/source/library/ssl_msg.c:3027: 0x58ae540: output record: msgtype = 23, version = [3:3], msglen = 32
/build/source/library/ssl_msg.c:3032: 0x58ae540: dumping 'output record sent to network' (37 bytes)
/build/source/library/ssl_msg.c:3032: 0x58ae540: 0000:  17 03 03 00 20 a1 5b 4f f6 80 4c a7 b9 c7 38 29  .... .[O..L...8)
/build/source/library/ssl_msg.c:3032: 0x58ae540: 0010:  9b 68 89 3c 85 32 8e 75 31 74 d1 1a 2a 36 01 b3  .h.<.2.u1t..*6..
/build/source/library/ssl_msg.c:3032: 0x58ae540: 0020:  28 cc b5 93 f9                                   (....
/build/source/library/ssl_msg.c:2353: 0x58ae540: => flush output
/build/source/library/ssl_msg.c:2367: 0x58ae540: message length: 37, out_left: 37
/build/source/library/ssl_msg.c:2374: 0x58ae540: ssl->f_send() returned 37 (-0xffffffdb)
/build/source/library/ssl_msg.c:2401: 0x58ae540: <= flush output
/build/source/library/ssl_msg.c:3080: 0x58ae540: <= write record
/build/source/library/ssl_msg.c:5180: 0x58ae540: <= send alert message
/build/source/library/ssl_msg.c:6198: 0x58ae540: <= write close notify
/build/source/library/ssl_tls.c:5521: 0x58ae540: => free
/build/source/library/ssl_tls.c:5583: 0x58ae540: <= free
viric commented 1 month ago

With a different error, mbedtls also fails to connect, for example, to live-api-s.facebook.com:443, that also uses TLS1.3.

/build/source/library/ssl_msg.c:3297: 0x7fadb82c77b0: TLS handshake fragmentation not supported
/build/source/library/ssl_msg.c:4244: 0x7fadb82c77b0: mbedtls_ssl_handle_message_type() returned -28800 (-0x7080)
/build/source/library/ssl_tls13_client.c:2367: 0x7fadb82c77b0: mbedtls_ssl_read_record() returned -28800 (-0x7080)
/build/source/library/ssl_tls13_client.c:2547: 0x7fadb82c77b0: <= parse certificate request
/build/source/library/ssl_tls.c:4617: 0x7fadb82c77b0: <= handshake
info: RTMP_Connect1, TLS_Connect failed: -0x7080
/build/source/library/ssl_msg.c:6187: 0x7fadb82c77b0: => write close notify
/build/source/library/ssl_msg.c:6198: 0x7fadb82c77b0: <= write close notify
/build/source/library/ssl_tls.c:5521: 0x7fadb82c77b0: => free
/build/source/library/ssl_tls.c:5583: 0x7fadb82c77b0: <= free
gilles-peskine-arm commented 1 month ago

Please check https://github.com/Mbed-TLS/mbedtls/issues/9223 to see if you've encountered one of the known issues.