Error handling in TLS is incorrect or missing

[tomato42]

Describe the bug When malformed key shares are sent to the server, the server doesn't abort the connection or aborts it with wrong alerts

To Reproduce

reproducer

``` openssl req -x509 -newkey rsa -keyout /tmp/localhost.key -out /tmp/localhost.crt -subj /CN=localhost -nodes -batch openssl s_server -key /tmp/localhost.key -cert /tmp/localhost.crt -www -curves X25519MLKEM768:SecP256r1MLKEM768 2>server.err >server.out & openssl_pid=$! git clone https://github.com/tomato42/tlsfuzzer pushd tlsfuzzer # won't be needed after https://github.com/tlsfuzzer/tlsfuzzer/pull/968 is merged git checkout full-fuzz-for-mlkem-share git clone https://github.com/tomato42/tlslite-ng .tlslite-ng ln -s .tlslite-ng/tlslite tlslite git clone https://github.com/warner/python-ecdsa .python-ecdsa ln -s .python-ecdsa/src/ecdsa ecdsa git clone https://github.com/GiacomoPope/kyber-py .kyber-py ln -s .kyber-py/src/kyber_py kyber_py PYTHONPATH=. python scripts/test-tls13-mlkem.py --no-fuzz --kems x25519mlkem768,secp256r1mlkem768 popd kill $openssl_pid ```

OpenSSL output

``` 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: 8022C3703E7F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:689: 8022C3703E7F0000:error:0800006B:elliptic curve routines:EC_POINT_set_affine_coordinates:point is not on curve:crypto/ec/ec_lib.c:879: 8022C3703E7F0000:error:0A0C0103:SSL routines:ssl_encapsulate:internal error:ssl/s3_lib.c:5006: 8022C3703E7F0000:error:1C8000A4:Provider routines:ossl_ecx_compute_key:failed during derivation:crypto/ec/ecx_key.c:143: 8022C3703E7F0000:error:0A0C0103:SSL routines:ssl_encapsulate:internal error:ssl/s3_lib.c:5006: 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: 8022C3703E7F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:689: 8022C3703E7F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:689: 8022C3703E7F0000:error:0A000065:SSL routines:final_key_share:no suitable key share:ssl/statem/extensions.c:1470: 8022C3703E7F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:689: 8022C3703E7F0000:error:0A000132:SSL routines:tls_parse_ctos_key_share:bad ecpoint:ssl/statem/extensions_srvr.c:693: ```

tlsfuzzer output

``` sanity ... OK x25519mlkem768: padded key share ... OK secp256r1mlkem768: basic connection ... OK x25519mlkem768: malformed pqc part, variable 1 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) x25519mlkem768: malformed pqc part, variable 2 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: malformed classical part ... Error encountered while processing node ExpectAlert(level=2, description=(47,)) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 242, in run node.process(self.state, msg) File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/expect.py", line 1973, in process raise AssertionError(problem_desc) AssertionError: Expected alert description "illegal_parameter" does not match received "internal_error" x25519mlkem768: basic connection with HRR ... OK x25519mlkem768: malformed classical part ... Error encountered while processing node ExpectAlert(level=2, description=(47,)) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 242, in run node.process(self.state, msg) File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/expect.py", line 1973, in process raise AssertionError(problem_desc) AssertionError: Expected alert description "illegal_parameter" does not match received "internal_error" secp256r1mlkem768: invalid ECDH point format: raw ... OK secp256r1mlkem768: invalid ECDH point format: hybrid ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) x25519mlkem768: truncated key share ... OK secp256r1mlkem768: padded key share ... OK secp256r1mlkem768: truncated key share ... OK x25519mlkem768: malformed pqc part, variable 0 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: basic connection with HRR ... OK x25519mlkem768: malformed pqc part, variable 3 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: malformed pqc part, variable 2 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: malformed pqc part, variable 1 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp384r1mlkem1024: basic connection ... OK x25519mlkem768: basic connection ... OK secp256r1mlkem768: malformed pqc part, variable 0 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: malformed pqc part, variable 3 ... Error encountered while processing node ExpectAlert(level=2, description=47) (child: ) with last message being: Error while processing Traceback (most recent call last): File "/home/hkario/dev/tlsfuzzer/scripts/test-tls13-mlkem.py", line 454, in main runner.run() File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 235, in run raise AssertionError("Unexpected message from peer: " + AssertionError: Unexpected message from peer: Handshake(server_hello) secp256r1mlkem768: invalid ECDH point format: compressed ... OK sanity ... OK Test hybrid ML-KEM key share support Verify that hybrid ML-KEM key exchange groups are supported and basic error handling is implemented correctly. Test end ==================== version: 4 ==================== TOTAL: 24 SKIP: 0 PASS: 13 XFAIL: 0 FAIL: 11 XPASS: 0 ==================== FAILED: 'secp256r1mlkem768: invalid ECDH point format: hybrid' 'secp256r1mlkem768: malformed classical part' 'secp256r1mlkem768: malformed pqc part, variable 0' 'secp256r1mlkem768: malformed pqc part, variable 1' 'secp256r1mlkem768: malformed pqc part, variable 2' 'secp256r1mlkem768: malformed pqc part, variable 3' 'x25519mlkem768: malformed classical part' 'x25519mlkem768: malformed pqc part, variable 0' 'x25519mlkem768: malformed pqc part, variable 1' 'x25519mlkem768: malformed pqc part, variable 2' 'x25519mlkem768: malformed pqc part, variable 3' ```

Expected behavior

1. The server should check validity of the client encapsulation key, failure in that should generate illegal_parameter alert
2. The server should check the validity of the client ECDH key share, failure in that should generate illegal_parameter alert
3. The server should validate that the client ECDH key share is in the uncompressed point format, it should reject shares in hybrid point format

Screenshots n/a

Environment (please complete the following information):

• OS: RHEL-10
• OpenSSL version: 3.2.2
• oqsprovider version: 0.7.1-dev
• liboqs version: 0.11.0

[beldmit]

I'm not sure that this is a oqs provider bug. I would consider it as oqs provider bug if the provider caught an error and didn't add it to the error stack, otherwise it looks like an OpenSSL issue.

I'm also sure that wrong alert is definitely an OpenSSL issue because alerts are thrown at OpenSSL level

[tomato42]

Alt least the lack of encapsulation key checks is in liboqs, and will need to be handled in oqsprovider: https://github.com/open-quantum-safe/liboqs/issues/1951

[tomato42]

@beldmit oh, and I suspect that OpenSSL knows how to reject the hybrid point encoding, I suspect it's just not configured by oqsprovider to do it

[beldmit]

@tomato42 could you please also duplicate this issue to OpenSSL?

[tomato42]

@beldmit done