TLS1.3 with brainpool version incompatibility

[bernd-edlinger]

When an openssl-3.0 client wants to support brainpool server certificates, but also allow TLS1.3 with RSA certificates, everything works fine, when both client and server enable brainpool groups, in addition to nist groups. So the client has to use this command line: ./openssl s_client -groups P-256:brainpoolP256r1 and the openssl-3.0 server uses this command line for brainpool: ./openssl s_server -cert ../test/certs/server-ecdsa-brainpoolP256r1-cert.pem -key ../test/certs/server-ecdsa-brainpoolP256r1-key.pem -groups P-256:brainpoolP256r1 or this command line for RSA: ./openssl s_server everything works fine, RSA use TLS1.3 and ECC use TLS1.2. Also if both server and client use openssl-3.2 everything works as well, except the ECC use TLS1.3, as expected. BUT the openssl-3.0 client cannot connect to the openssl-3.2 server when the server uses brainpool:

$ ./openssl s_client -groups P-256:brainpoolP256r1 -trace
CONNECTED(00000003)
Sent Record
Header:
  Version = TLS 1.0 (0x301)
  Content Type = Handshake (22)
  Length = 327
    ClientHello, Length=323
      client_version=0x303 (TLS 1.2)
      Random:
        gmt_unix_time=0xD3B13943
        random_bytes (len=28): 24530A97534115EA57B17C77ABAC33F909733FDBAA73C61AD594C525
      session_id (len=32): 397A7086775ACCDB82A75E2B19B7FE6459BF3088B322A7A5D8EAC17791A76D40
      cipher_suites (len=62)
        {0x13, 0x02} TLS_AES_256_GCM_SHA384
        {0x13, 0x03} TLS_CHACHA20_POLY1305_SHA256
        {0x13, 0x01} TLS_AES_128_GCM_SHA256
        {0xC0, 0x2C} TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        {0xC0, 0x30} TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        {0x00, 0x9F} TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        {0xCC, 0xA9} TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xA8} TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xAA} TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xC0, 0x2B} TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x2F} TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        {0x00, 0x9E} TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x24} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        {0xC0, 0x28} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
        {0x00, 0x6B} TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        {0xC0, 0x23} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        {0xC0, 0x27} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x67} TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        {0xC0, 0x0A} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        {0xC0, 0x14} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x39} TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        {0xC0, 0x09} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        {0xC0, 0x13} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x33} TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x9D} TLS_RSA_WITH_AES_256_GCM_SHA384
        {0x00, 0x9C} TLS_RSA_WITH_AES_128_GCM_SHA256
        {0x00, 0x3D} TLS_RSA_WITH_AES_256_CBC_SHA256
        {0x00, 0x3C} TLS_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x35} TLS_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x2F} TLS_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0xFF} TLS_EMPTY_RENEGOTIATION_INFO_SCSV
      compression_methods (len=1)
        No Compression (0x00)
      extensions, length = 188
        extension_type=server_name(0), length=14
          0000 - 00 0c 00 00 09 6c 6f 63-61 6c 68 6f 73 74      .....localhost
        extension_type=ec_point_formats(11), length=4
          uncompressed (0)
          ansiX962_compressed_prime (1)
          ansiX962_compressed_char2 (2)
        extension_type=supported_groups(10), length=6
          secp256r1 (P-256) (23)
          brainpoolP256r1 (26)
        extension_type=session_ticket(35), length=0
        extension_type=encrypt_then_mac(22), length=0
        extension_type=extended_master_secret(23), length=0
        extension_type=signature_algorithms(13), length=42
          ecdsa_secp256r1_sha256 (0x0403)
          ecdsa_secp384r1_sha384 (0x0503)
          ecdsa_secp521r1_sha512 (0x0603)
          ed25519 (0x0807)
          ed448 (0x0808)
          rsa_pss_pss_sha256 (0x0809)
          rsa_pss_pss_sha384 (0x080a)
          rsa_pss_pss_sha512 (0x080b)
          rsa_pss_rsae_sha256 (0x0804)
          rsa_pss_rsae_sha384 (0x0805)
          rsa_pss_rsae_sha512 (0x0806)
          rsa_pkcs1_sha256 (0x0401)
          rsa_pkcs1_sha384 (0x0501)
          rsa_pkcs1_sha512 (0x0601)
          ecdsa_sha224 (0x0303)
          rsa_pkcs1_sha224 (0x0301)
          dsa_sha224 (0x0302)
          dsa_sha256 (0x0402)
          dsa_sha384 (0x0502)
          dsa_sha512 (0x0602)
        extension_type=supported_versions(43), length=9
          TLS 1.3 (772)
          TLS 1.2 (771)
          TLS 1.1 (770)
          TLS 1.0 (769)
        extension_type=psk_key_exchange_modes(45), length=2
          psk_dhe_ke (1)
        extension_type=key_share(51), length=71
            NamedGroup: secp256r1 (P-256) (23)
            key_exchange:  (len=65): 04B9F00C469BB69090E6B7E8FC0EF518A600EEA9950825CFAFBC88DCF2B0EE218100ABD95B8C6CFAA29A7302332E42FCB07286C0653C823E5896981ED97C363D0D

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Alert (21)
  Length = 2
    Level=fatal(2), description=handshake failure(40)

40778BA11E7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1605:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 332 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Reason is the openssl-3.2 server chooses TLS1.3 although there is no signature algorithm ecdsa_brainpoolP256r1_sha256 advertised by the openssl-3.0 client, so that is bound to fail. As a workaround either the openssl-3.0 client or the openssl-3.2 server could disable tls1_3 but that is not desirable for the RSA server.

[t8m]

If both versions support TLS-1.3 they must never choose older version. If they cannot agree on supported groups connection failure is inevitable.

[bernd-edlinger]

The problem is the client cannot know how the server is configured, and therefore the server should take the signature_algorithms extension into account, because the meaning of the ECDSA signature algorithm IDs are quite different for TLS1.3 and for TLS1.2: TLS1.3 implies a 1:1 connection between ECDSA signature algorithms and EC groups, but TLS1.2 allows the cross product of signature_algorithms extension and supported_groups extension. The server should know that, and choose TLS1.2 if it has the freedom to do so.

[kroeckx]

If both sides support TLS 1.3, that should be used. Otherwise you can force the use of older TLS versions.

[bernd-edlinger]

What you say means that, when the server does only support TLS1.2, then a man in the middle could simply remove any signature that he don't like from the signature_algorithms extension of the client hello, say everything with sha256, sha384 and sha512 in the name, and neither the server nor the client will notice anything from this attack?

Please say that there is some kind of a posteriori authentication of the unencrypted handshake messages exchanged between client and server, even if the TLS1.2 protocol is used.

[mattcaswell]

Please say that there is some kind of a posteriori authentication of the unencrypted handshake messages exchanged between client and server, even if the TLS1.2 protocol is used.

The entire handshake is authenticated in the Finished message.

[bernd-edlinger]

The entire handshake is authenticated in the Finished message.

Thanks for the confirmation. I want to use the received client hello's signature_algorithm extension in addition to the server's configured -sigalgs in is_tls13_capable to make the right decision which protocol to use.

[mattcaswell]

If the server is configured in a way that TLSv1.3 is possible, and the client claims TLSv1.3 support then TLSv1.3 should be negotiated. Even if that means that the handshake later fails because the client and server cannot agree on sigalgs/ciphersuites/whatever.

[bernd-edlinger]

But, because the client does advertise brainpool in the groups extension but not in the signature_algorithm extension, it is clear that it does simply not implement those and, expects the server to use TLS1.2 only in this case, and TLS1.3 should of course be used for RSA certificates.

[mattcaswell]

it is clear that it does simply not implement those and, expects the server to use TLS1.2 only in this case

I'm failing to see the issue here. The client support TLSv1.3. It doesn't support brainpool sigalgs. The server supports TLSv1.3 and only support brainpool sigalgs. The correct outcome is TLSv1.3 is chosen and then handshake failure results because of the failure to agree on a sigalg. This is the expected outcome.

[bernd-edlinger]

Okay, then let me summarize how I see the situation: I have a software that consists of client and server which is distributed over the internet. The servers can use either RSA, nist, ED25519, ED448 or brainpool certificate. The clients dont know in advance which certificate type will be used. Both client and server are prepared for this situation in this way: They call SSL_CTX_set1_curves with all required curves, but they do not try to interfere with the used signature_algorithms, so there is no call to SSL_CTX_set1_sigalgs_list. Currently everything is working, but they are currently still using openssl-3.0. And as soon as the servers will be updated to openssl-3.2 some connections will start to fail. If I see that right, there will not even be a re-compilation necessary, just replacing the shared objects will be enough to create unexpected incompatibilities between some servers and some clients. This is ABSOLUTELY NOT the expected outcome.

[bernd-edlinger]

So as a compromise how about that:

Instead of looking at the signature_algorithm extension of the client hello, I propose to do this in tls_check_sigalg_curve:

In case of a brainpool curves, if TLS1.2 is not disabled, and the supported groups contain only the TLS1.2 brainpool id, but the TLS1.3 brainpool sigalg is only from the default sigalgs list, and was not explicitly specified, it is clear that this configuration is intended to be useful for TLS1.2, so don't try to negotiate TLS1.3 which might fail.

[mattcaswell]

Finally, I think I am starting to understand the problem you are describing. Sorry for being a bit slow.

Let me describe it in my own words:

In 3.0 there was no support for brainpool certs in TLSv1.3 (but they were supported in TLSv1.2). A server configured with only a brainpool cert runs the "is_tls13_capable" check, discovers that it only has TLSv1.2 capable certs and therefore will only negotiate TLSv1.2 as a maximum

A 3.0 client (which has TLSv1.3 enabled but has no TLSv1.3 brainpool sigalgs support) connects to such a server and the connection is successfully established using TLSv1.2

The server operator then upgrades to 3.2 but does not otherwise change their configuration. In 3.2 we added TLSv1.3 brainpool sigalg support. The server is configured with a brainpool cert and runs the "is_tls13_capable" check. Since it now supports TLSv1.3 with that cert TLSv1.3 is now enabled (where it wasn't prior to the upgrade).

The same 3.0 client (still with TLSv1.3 enabled, but still with no TLSv1.3 brainpool sigalgs support) connects to the server. Since both client and server both have TLSv1.3 enabled, TLSv1.3 is selected. Since the client doesn't have the brainpool sigalgs support the handshake fails.

So, this represents an ABI break because the only change here is that the server upgraded from 3.0 to 3.2 and suddenly handshakes start failing.

Is that correct?

[bernd-edlinger]

Is that correct?

Yes, exactly.

[bernd-edlinger]

A possible fix for the ABI break is in #25733