Kestrel HTTPS: Incorrect behavior when missing access to private key on Windows

[OronDF343]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

When using an HTTPS endpoint in Kestrel with a certificate from the Windows LocalMachine certificate store, if the ASP.NET Core application is running without administrator privileges, it usually does not have access to the private key of the certificate, only the public information.

If this is the case, Kestrel will start successfully, but any attempt to access the service over HTTPS will result in a "connection closed" error (ERR_CONNECTION_CLOSED in Chromium, PR_END_OF_FILE_ERROR in Firefox). Kestrel does not log an error, except at the Debug log level, while most other certificate issues will cause the application to fail to start entirely.

Aside from the exception logged at the Debug level, the only other indication that something is amiss is in the event viewer, where a generic error is logged by Schannel every time Kestrel attempts to load the private key, however no information about the application is present in these errors.

This issue can be very difficult to diagnose on Windows Server when running an ASP.NET Core app as a Windows Service running under a non-admin user.

Expected Behavior

If possible, the application should fail to start with an exception, as happens with other certificate issues.

Otherwise, the exception thrown internally should be logged at a higher severity and with a clearer message.

Steps To Reproduce

These steps assume that UAC is enabled on the machine.

1. Open a new PowerShell window with administrator privileges and execute the following command to generate a certificate:

   New-SelfSignedCertificate -Subject TestCert -DnsName localhost -CertStoreLocation Cert:\LocalMachine\My

2. Open any terminal as a regular user in an empty directory and create a new ASP.NET Core Web API project:

   dotnet new webapi

3. Edit appsettings.Development.json and add the following section:

   "Kestrel": {
   "Endpoints": {
     "Https": {
       "Url": "https://localhost:7127",
       "Certificate": {
         "Store": "My",
         "Location": "LocalMachine",
         "Subject": "TestCert",
         "AllowInvalid": true
       }
     }
   }
   }

   Optionally change the LogLevel. NOTE: The value of AllowInvalid has no effect on this issue. I have set it to true in this example simply to avoid having to add the certificate to the root store.

4. Run the app with dotnet run and attempt to access https://localhost:7127/swagger in the browser. Observe the command line output and Event Viewer.

Exceptions (if any)

Error in Event Viewer:

A fatal error occurred when attempting to access the TLS server credential private key. The error code returned from the cryptographic module is 0x8009030D. The internal error state is 10001.

Exception logged by Kestrel:

dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1]
      Failed to authenticate HTTPS connection.
      System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognized
         at System.Net.SSPIWrapper.AcquireCredentialsHandle(ISSPIInterface secModule, String package, CredentialUse intent, SCH_CREDENTIALS* scc)
         at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCH_CREDENTIALS* secureCredential)
         at System.Net.Security.SslStreamPal.AcquireCredentialsHandleSchCredentials(SslAuthenticationOptions authOptions)
         at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions)
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions)
         at System.Net.Security.SslStream.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions)
         at System.Net.Security.SslStream.AcquireServerCredentials(Byte[]& thumbPrint)
         at System.Net.Security.SslStream.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output)
         at System.Net.Security.SslStream.NextMessage(ReadOnlySpan`1 incomingBuffer)
         at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize)
         at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken)
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)

.NET Version

7.0.400

Anything else?

For any others encountering this issue, here the correct way to fix the permissions on the private key:

1. Open certmgr.msc with admin permissions (start, find Manage computer certificates)
2. Locate the certificate that will be used
3. Right-click, All Tasks -> Manage Private Keys...
4. Add permissions for the user that will run the app

[amcasey]

@OronDF343 Those are great steps and I had no trouble reproducing the issue. Thanks!

Can you tell me a bit more about what you were hoping would happen in this scenario? I'm assuming this was happening in your development environment and not in production and increasing the logging level to Debug seems like a pretty natural investigatory step.

I'll investigate whether there's a way to do more validation of the certificate up-front, but there's a limit to how much overhead we want to add for an uncommon scenario (in the sense that you're likely to fix this the first time you hit it and then not see it again).

[amcasey]

It looks like we're at least trying to validate the accessibility of the private key: https://source.dot.net/#Microsoft.AspNetCore.Server.Kestrel.Core/CertificateLoader.cs,106

[OronDF343]

@amcasey This happened to two co-workers from different teams in production environments, on first installation of the services. Our default logging configurations for production does not print debug logs for the System and Microsoft namespaces. My involvement was to help solve the issue. (The reason why this wasn't the case every time a new service is created, is that most run with Local System privileges, which is bad, but the plan is to move to containers in the "near" future. Yes, our current processes and training need some improvement, sadly it is what it is)

I would have expected an unhandled exception to be thrown as usual, which is visible in event viewer, as is the case when the certificate does not exist, is invalid (when AllowInvalid = false), or the binding to the port fails (e.g. another service is listening on it or permission is denied), or when someone forgot to call UseWindowsService, or didn't set up DI correctly, etc.

I agree that validating on startup is not the ideal solution, since it is a one-time problem. I think that at the very least leaving the exception unhandled, or logging it with a higher severity would be a step in the right direction. Or ideally, wrapping it in another exception with a better description, because the current exception does not indicate clearly what the problem is (it mentions "Authentication" which can be confused with Authentication middleware, and it does not mention "certificate" or "private key" or "access denied").

[amcasey]

I agree that validating on startup is not the ideal solution, since it is a one-time problem. I think that at the very least leaving the exception unhandled, or logging it with a higher severity would be a step in the right direction. Or ideally, wrapping it in another exception with a better description, because the current exception does not indicate clearly what the problem is (it mentions "Authentication" which can be confused with Authentication middleware, and it does not mention "certificate" or "private key" or "access denied").

I believe that same exception can also indicate that a client authentication has occurred, so we'd need a way to detect that it was a failure of this particular sort. There's an inner exception, but it won't be the same on every platform.

My current plan is to pursue both options (up-front check and logging this particular failure mode more aggressively) and see which, if either, pans out.

[rsrokosz-bravurasolutions]

Hi, what's the latest on this issue? I think the behaviour is the same on .Net 8.

[pramahamuni]

I too facing the issue while upgrading the .net to .net 8 and this is the error i am getting did anyone faced similar issue ? Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:0A000412:SSL routines::sslv3 alert bad certificate --- End of inner exception stack trace --- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan1 input, Byte[]& sendBuf, Int32& sendCount) at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)