TLS 1.3 negotiation fails with FIPS-enabled Ubuntu 22.04 server and Chromium based browsers

[michaelwildvarian]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

When Kestrel is running on a FIPS-enabled Ubuntu 22.04, TLS 1.3 negotiation fails for Chromium browsers and ERR_SSL_PROTOCOL_ERROR is displayed.

Scenarios tested:

• Different clients against Kestrel server: Firefox, curl and OpenSSL s_client all work. Chrome and Chromium Edge fail.
• Different servers against Chrome: OpenSSL s_server, nginx and Python https.server+ssl work. Kestrel fails.

Expected Behavior

Chromium based browsers should be able to establish a TLS 1.3 session with Kestrel servers running on FIPS-enabled Ubuntu 22.04.

Steps To Reproduce

• Provision VM running Ubuntu 22.04
• Obtain a free Ubuntu Pro token by registering a private account at https://ubuntu.com/pro
• Enable FIPS on the VM:
  • sudo pro attach --no-auto-enable <token>
  • sudo pro enable fips-preview --assume-yes
  • sudo reboot
  • cat /proc/sys/crypto/fips_enabled <-- Ensure the output is 1
• Install ssl-cert package:
  • sudo apt install ssl-cert
• Install Chrome on the VM:
  • wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  • sudo dpkg -i google-chrome-stable_current_amd64.deb
  • Install .NET 8 SDK on the VM:
    • wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
    • sudo dpkg -i packages-microsoft-prod.deb
    • sudo apt update
    • sudo apt install dotnet-sdk-8.0
  • Configure a hostname and create a snakeoil certificate:
    • sudo bash -c 'echo some.host.internal > /etc/hostname
    • sudo bash -c 'echo "127.0.0.1 some.host.internal" >> /etc/hosts
    • sudo make-ssl-cert generate-default-snakeoil --force-overwrite
• Create a ASP.NET Core webapi boilerplate application
  • mkdir app
  • cd app
  • dotnet new webapi -au None
• Add the following configurations to appsetting.Development.json
  • Urls = https://127.0.0.1:4445
  • Kestrel:Certificates:Default:Path = /etc/ssl/certs/ssl-cert-snakeoil.pem
  • Kestrel:Certificates:Default:KeyPath = /etc/ssl/private/ssl-cert-snakeoil.key
  • Logging:LogLevel:Microsoft.AspeNetCore.HttpLogging.HttpLoggingMiddleware = Trace
  • Logging:LogLevel:Microsoft.AspeNetCore.Server.Kestrel = Trace
• Start the application (with sudo for access to the certificates)
  • dotnet build
  • sudo dotnet run --no-build
• Start the Chrome browser and navigate to https://some.host.internal:4445/weatherforecast. It should display the following:
• By forcing TLS 1.2 Chrome is able to connect:
  • Add Kestrel:EndpointDefaults:SslProtocols = [ "Tls12" ] to appsettings.Developmen.json
  • Restart the app
  • Reload the page in Chrome, accept the security risk of the self-signed certificate and Chrome should display the following:
• Observe that openssl s_server is able to successfully negotiate TLS 1.3 with Chrome:
  • sudo openssl s_server -key /etc/ssl/private/ssl-cert-snakeoil.key -cert /etc/ssl/certs/ssl-cert-snakeoil.pem -accept 4446 -www
  • Open https://some.host.internal:4446 in Chrome. It should display a successful TLS 1.3 connection (after accepting the self-signed certificate):

Exceptions (if any)

dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
      Hosting starting
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[4]
      Created directory watcher for '/etc/ssl/certs'.
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[5]
      Created file watcher for '/etc/ssl/certs/ssl-cert-snakeoil.pem'.
trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[11]
      Added observer to file watcher for '/etc/ssl/certs/ssl-cert-snakeoil.pem'.
trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[13]
      File '/etc/ssl/certs/ssl-cert-snakeoil.pem' now has 1 observers.
trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[14]
      Directory '/etc/ssl/certs' now has watchers on 1 files.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://127.0.0.1:4445
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/vendor/dotnettest
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
      Hosting started

dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39]
      Connection id "0HN1JEM94TIM8" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HN1JEM94TIM8" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6]
      Connection id "0HN1JEM94TIM8" received FIN.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39]
      Connection id "0HN1JEM94TIM9" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HN1JEM94TIM9" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6]
      Connection id "0HN1JEM94TIM9" received FIN.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39]
      Connection id "0HN1JEM94TIMA" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HN1JEM94TIMA" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
      System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
       ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol
         --- End of inner exception stack trace ---
         at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
         at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
      System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
       ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol
         --- End of inner exception stack trace ---
         at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
         at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
      System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
       ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol
         --- End of inner exception stack trace ---
         at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
         at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2]
      Connection id "0HN1JEM94TIM8" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2]
      Connection id "0HN1JEM94TIMA" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2]
      Connection id "0HN1JEM94TIM9" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HN1JEM94TIMA" sending FIN because: "The Socket transport's send loop completed gracefully."
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HN1JEM94TIM9" sending FIN because: "The Socket transport's send loop completed gracefully."
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39]
      Connection id "0HN1JEM94TIMB" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1]
      Connection id "0HN1JEM94TIMB" started.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HN1JEM94TIM8" sending FIN because: "The Socket transport's send loop completed gracefully."
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6]
      Connection id "0HN1JEM94TIMB" received FIN.
dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
      System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
       ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol
         --- End of inner exception stack trace ---
         at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
         at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2]
      Connection id "0HN1JEM94TIMB" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HN1JEM94TIMB" sending FIN because: "The Socket transport's send loop completed gracefully."

.NET Version

8.0.200

Anything else?

Output of dotnet --info:

.NET SDK:
 Version:           8.0.200
 Commit:            438cab6a9d
 Workload version:  8.0.200-manifests.cdf2cc8e

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.200/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.2
  Architecture: x64
  Commit:       1381d5ebd2

.NET SDKs installed:
  8.0.200 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

On request I can provide Wireshark captures via email.

[michaelwildvarian]

As an addendum: NIST SP 800-52 R2 clearly states that systems shall support TLS 1.3 by January 1 2024.

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Is there an existing issue for this? - [X] I have searched the existing issues ### Describe the bug When Kestrel is running on a FIPS-enabled Ubuntu 22.04, TLS 1.3 negotiation fails for Chromium browsers and `ERR_SSL_PROTOCOL_ERROR` is displayed. **Scenarios tested:** * Different clients against Kestrel server: Firefox, curl and OpenSSL s_client all work. Chrome and Chromium Edge fail. * Different servers against Chrome: OpenSSL s_server, nginx and Python https.server+ssl work. Kestrel fails. ### Expected Behavior Chromium based browsers should be able to establish a TLS 1.3 session with Kestrel servers running on FIPS-enabled Ubuntu 22.04. ### Steps To Reproduce * Provision VM running Ubuntu 22.04 * Obtain a free Ubuntu Pro token by registering a private account at https://ubuntu.com/pro * Enable FIPS on the VM: * `sudo pro attach --no-auto-enable ` * `sudo pro enable fips-preview --assume-yes` * `sudo reboot` * `cat /proc/sys/crypto/fips_enabled` <-- Ensure the output is `1` * Install ssl-cert package: * `sudo apt install ssl-cert` * Install Chrome on the VM: * `wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb` * `sudo dpkg -i google-chrome-stable_current_amd64.deb` * Install .NET 8 SDK on the VM: * `wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb` * `sudo dpkg -i packages-microsoft-prod.deb` * `sudo apt update` * `sudo apt install dotnet-sdk-8.0` * Configure a hostname and create a snakeoil certificate: * `sudo bash -c 'echo some.host.internal > /etc/hostname` * `sudo bash -c 'echo "127.0.0.1 some.host.internal" >> /etc/hosts` * `sudo make-ssl-cert generate-default-snakeoil --force-overwrite` * Create a ASP.NET Core webapi boilerplate application * `mkdir app` * `cd app` * `dotnet new webapi -au None` * Add the following configurations to `appsetting.Development.json` * `Urls = https://127.0.0.1:4445` * `Kestrel:Certificates:Default:Path = /etc/ssl/certs/ssl-cert-snakeoil.pem` * `Kestrel:Certificates:Default:KeyPath = /etc/ssl/private/ssl-cert-snakeoil.key` * `Logging:LogLevel:Microsoft.AspeNetCore.HttpLogging.HttpLoggingMiddleware = Trace` * `Logging:LogLevel:Microsoft.AspeNetCore.Server.Kestrel = Trace` * Start the application (with sudo for access to the certificates) * `dotnet build` * `sudo dotnet run --no-build` * Start the Chrome browser and navigate to `https://some.host.internal:4445/weatherforecast`. It should display the following:  * By forcing TLS 1.2 Chrome is able to connect: * Add `Kestrel:EndpointDefaults:SslProtocols = [ "Tls12" ]` to `appsettings.Developmen.json` * Restart the app * Reload the page in Chrome, accept the security risk of the self-signed certificate and Chrome should display the following:  * Observe that `openssl s_server` is able to successfully negotiate TLS 1.3 with Chrome: * `sudo openssl s_server -key /etc/ssl/private/ssl-cert-snakeoil.key -cert /etc/ssl/certs/ssl-cert-snakeoil.pem -accept 4446 -www` * Open https://some.host.internal:4446 in Chrome. It should display a successful TLS 1.3 connection (after accepting the self-signed certificate):  ### Exceptions (if any) ``` dbug: Microsoft.Extensions.Hosting.Internal.Host[1] Hosting starting dbug: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[4] Created directory watcher for '/etc/ssl/certs'. dbug: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[5] Created file watcher for '/etc/ssl/certs/ssl-cert-snakeoil.pem'. trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[11] Added observer to file watcher for '/etc/ssl/certs/ssl-cert-snakeoil.pem'. trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[13] File '/etc/ssl/certs/ssl-cert-snakeoil.pem' now has 1 observers. trce: Microsoft.AspNetCore.Server.Kestrel.Core.Internal.CertificatePathWatcher[14] Directory '/etc/ssl/certs' now has watchers on 1 files. info: Microsoft.Hosting.Lifetime[14] Now listening on: https://127.0.0.1:4445 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development info: Microsoft.Hosting.Lifetime[0] Content root path: /home/vendor/dotnettest dbug: Microsoft.Extensions.Hosting.Internal.Host[2] Hosting started dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39] Connection id "0HN1JEM94TIM8" accepted. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1] Connection id "0HN1JEM94TIM8" started. dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6] Connection id "0HN1JEM94TIM8" received FIN. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39] Connection id "0HN1JEM94TIM9" accepted. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1] Connection id "0HN1JEM94TIM9" started. dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6] Connection id "0HN1JEM94TIM9" received FIN. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39] Connection id "0HN1JEM94TIMA" accepted. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1] Connection id "0HN1JEM94TIMA" started. dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1] Failed to authenticate HTTPS connection. System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol --- End of inner exception stack trace --- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context) dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1] Failed to authenticate HTTPS connection. System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol --- End of inner exception stack trace --- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context) dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1] Failed to authenticate HTTPS connection. System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol --- End of inner exception stack trace --- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context) dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HN1JEM94TIM8" stopped. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HN1JEM94TIMA" stopped. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HN1JEM94TIM9" stopped. dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HN1JEM94TIMA" sending FIN because: "The Socket transport's send loop completed gracefully." dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HN1JEM94TIM9" sending FIN because: "The Socket transport's send loop completed gracefully." dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39] Connection id "0HN1JEM94TIMB" accepted. dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1] Connection id "0HN1JEM94TIMB" started. dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HN1JEM94TIM8" sending FIN because: "The Socket transport's send loop completed gracefully." dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6] Connection id "0HN1JEM94TIMB" received FIN. dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1] Failed to authenticate HTTPS connection. System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000EB:SSL routines::no application protocol --- End of inner exception stack trace --- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context) dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HN1JEM94TIMB" stopped. dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HN1JEM94TIMB" sending FIN because: "The Socket transport's send loop completed gracefully." ``` ### .NET Version 8.0.200 ### Anything else? Output of `dotnet --info`: ``` .NET SDK: Version: 8.0.200 Commit: 438cab6a9d Workload version: 8.0.200-manifests.cdf2cc8e Runtime Environment: OS Name: ubuntu OS Version: 22.04 OS Platform: Linux RID: linux-x64 Base Path: /usr/share/dotnet/sdk/8.0.200/ .NET workloads installed: There are no installed workloads to display. Host: Version: 8.0.2 Architecture: x64 Commit: 1381d5ebd2 .NET SDKs installed: 8.0.200 [/usr/share/dotnet/sdk] .NET runtimes installed: Microsoft.AspNetCore.App 8.0.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 8.0.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App] Other architectures found: None Environment variables: Not set global.json file: Not found Learn more: https://aka.ms/dotnet/info Download .NET: https://aka.ms/dotnet/download ``` On request I can provide Wireshark captures via email.

Author:	michaelwildvarian
Assignees:	-
Labels:	`area-System.Net.Security`, `untriaged`
Milestone:	-

[amcasey]

This seems like it might be a question for https://github.com/dotnet/runtime. Thoughts, @CarnaViire @wfurt?

[amcasey]

@michaelwildvarian Does chromium work on a non-FIPS-enabled Ubuntu 22.04 server?

Naively, this would be my first guess about what's going wrong: https://learn.microsoft.com/en-us/dotnet/core/extensions/sslstream-troubleshooting#client-and-server-do-not-possess-a-common-algorithm. I'd expect this to be visible in the wireshark trace.

[michaelwildvarian]

@amcasey It also fails if Chromium is running on a non-FIPS client. But I'll check the article you linked and will get back to you.

[michaelwildvarian]

@amcasey, doesn't look like this is the exact issue. What I see in Wireshark:

1. Client Hello

2. Hello Retry Request

   Strangely, the retry request sent by the server contains the second cipher offered by the client (0x1302, TLS_AES_256_GCM_SHA384).

3. Change Cipher Spec

4. Fatal Alert (No application protocol)

If I do the same with openssl s_server as server, the Hello Retry Request specifies TLS_AES_128_GCM_SHA256 (0x1301) instead, to which the client answers with the same Change Cipher Spec message and the s_server then happily proceeds with the Server Hello and TLS_AES_128_GCM_SHA256 is used.

So, the problem appears to be something with to do with TLS_AES_256_GCM_SHA384. But both client and server claim support, so why does the server reject?

[rzikm]

On request I can provide Wireshark captures via email.

@michaelwildvarian Can you send the captures to redacted (at) microsoft.com?

Edit: removed the address now that I got the mail

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[wfurt]

I'm wondering if the issue is that we fall-back to the .NET set @rzikm https://github.com/dotnet/runtime/issues/98797#issuecomment-1964553152

[michaelwildvarian]

@rzikm Just sent the mail.

[michaelwildvarian]

I'm wondering if the issue is that we fall-back to the .NET set @rzikm #98797 (comment)

@wfurt, let me try this. Indeed the default /etc/ssl/openssl.cnf does not explicitly list any ciphers, it just contains the defaults:

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=2

I tried with replacing the cipher string with the output of openssl ciphers -s. I also set the OPENSSL_CONF=/etc/ssl/openssl.cnf environment variable to make sure to pick up the correct file. But no joy.

[vcsjones]

DOTNET_DEFAULT_CIPHERSTRING

This cipher list is the default for SSL_CTX_set_cipher_list, and that OpenSSL API is only for TLS v1.2 and below.

[rzikm]

Got it, the trigger is when server issues HelloRetryRequest (e.g. when client does not use preferred group in key_share), when we attempt to do ALPN negotiation the second time, the list of ALPNs is no longer there

https://github.com/dotnet/runtime/blob/71f3bf163b2ab8ea297350dfcc2362a27ec439ee/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs#L683-L696

We should look at it in 9.0 timeframe.

[michaelwildvarian]

Many thanks for the analysis. Is this something we can work around?

[rzikm]

It is possible to tweak the server preferred groups via openssl.cnf

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Groups = 

Not sure what the best value for the Groups field is, Chrome is attempting x25519, your server wants secp256r1, so it should probably include both of those (e.g. x25519:secp256r1), or perhaps x25519:DEFAULT.

This will make server accept the curve and not issue HelloRetryRequest, but perhaps there is a reason why x25519 is not accepted when fips-preview is applied. I defer to @bartonjs and @vcsjones's judgement as to what is a good setting here.

[vcsjones]

there is a reason why x25519 is not accepted when fips-preview is applied

X25519 is not a FIPS approved algorithm. When you run in FIPS mode, the algorithm gets turned off.

From FIPS 140-3-IG:

The publication of SP 800-186 does not impact the curves permitted under SP 800-56Arev3. Curves that are included in SP 800-186 but not included in SP 800-56Arev3 are not approved for key agreement. E.g., the ECDH X25519 and X448 key agreement schemes (defined in RFC 7748) that use Curve25519 and Curve448, respectively, are not compliant to SP 800-56Arev3.

[rzikm]

Reopening for backport consideration.

[rzikm]

@michaelwildvarian does this issue prevent you from migrating to .NET8? It makes it easier for us to get the servicing approved if we have business justification.

[michaelwildvarian]

@rzikm , we are already on .NET8 but have design freeze somewhere in April, early May. So far we have TLS 1.3 disabled (by configuration), but it would be great to release the product with TLS 1.3 support, so we don't have to do a patch release before it even hits the market.

[rzikm]

I see, and FIPS is somehow required for your scenario?

In case you don't want to disclose something publicly here at GitHub, you can contact me privately at radekzikmund [at] microsoft.com.

[KenMcC98]

@rzikm the issue also affects us (we had duplicate issue dotnet/aspnetcore/54366 raised) and is preventing us fully rolling out a firm-wide .NET 8 base image.

Our enterprise runs OpenShift clusters that have FIPS mandated for all accounts, so anyone running the image would have to follow the workaround.

[michaelwildvarian]

@KenMcC98 just to point out (and be a stickler...): The workaround breaks FIPS compliance, so it really isn't one.

[michaelwildvarian]

@rzikm many thanks for getting the fix into the 8.0.5 milestone! 🙏

[karelz]

Fixed in main (9.0) in PR #99357 and in 8.0.5 in PR #99670.