HttpClient not sending certificate setup in handler

[eeaquino]

Description

When going from .Net 8 to .Net 9 and setting up a Named HttpClient for DI with a handler the certificate is never sent. Verified the certificate is never sent by using WireShark to intercept the network communications.

Reproduction Steps

.Net 8

services.AddHttpClient("NamedClient",
        client => { client.BaseAddress = new Uri("serverUrl"); })
    .ConfigurePrimaryHttpMessageHandler(() =>
        {
            var handler = new HttpClientHandler
            {
                ClientCertificateOptions = ClientCertificateOption.Manual
            };

            var certs = new X509Certificate2(
                configuration.GetValue<string>("CertPath")!,
                configuration.GetValue<string>("CertPassword")!, X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.MachineKeySet);
            handler.ClientCertificates.Add(certs);
            return handler;
        }
    )

.Net 9, Also tired with the same Obsolete code using new X509Certificate2 with the same result.

services.AddHttpClient("NamedClient",
        client => { client.BaseAddress = new Uri("serverUrl"); })
    .ConfigurePrimaryHttpMessageHandler(() =>
        {
            var handler = new HttpClientHandler
            {
                ClientCertificateOptions = ClientCertificateOption.Manual
            };

            var certs =X509CertificateLoader.LoadPkcs12FromFile(
                configuration.GetValue<string>("CertPath")!,
                configuration.GetValue<string>("CertPassword")!, X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.MachineKeySet);
            handler.ClientCertificates.Add(certs);
            return handler;
        }
    )

Expected behavior

Certificate is sent during handshake.

Actual behavior

No certificate sent

Regression?

Worked as Expected .Net 8

Known Workarounds

No response

Configuration

.Net 9 RC2 Windows 11 x64

Other information

No response

[ManickaP]

@CarnaViire is it possible that this is related to your Keyed DI changes in .NET 9?

[CarnaViire]

@CarnaViire is it possible that this is related to your Keyed DI changes in .NET 9?

I doubt it, looks more like a certificate handling regression... might be related to https://github.com/dotnet/runtime/issues/109007

@eeaquino if instead of HttpClientFactory you'd use a static or singleton HttpClient, do you see the same regression?

E.g.

services.AddKeyedSingleton<HttpClient>("NamedClient", (_, _) => 
{
    var handler = new HttpClientHandler { ClientCertificateOptions = ClientCertificateOption.Manual };

    var certs = new X509Certificate2(
        configuration.GetValue<string>("CertPath")!,
        configuration.GetValue<string>("CertPassword")!,
        X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.MachineKeySet);
   handler.ClientCertificates.Add(certs);

   return new HttpClient(handler) { BaseAddress = new Uri("serverUrl") };
});

// -----

public class MyService
{
    // public MyService(IHttpClientFactory httpClientFactory, ...
    public MyService([FromKeyedServices("NamedClient")] HttpClient httpClient, // ...

/cc @rzikm @wfurt for cert-related insights

[rzikm]

I don't think there was any significant change we handle certificates in SocketsHttpHandler between .NET 8 and .NET 9. can you check the workaround suggested in https://github.com/dotnet/runtime/issues/109007#issuecomment-2422836153?

[eeaquino]

I tested both proposed workarounds, singleton and adding loader limits with no success.

exception System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> 
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> 
System.ComponentModel.Win32Exception (0x8009030D): The credentials supplied to the package were not recognized at 
System.Net.SSPIWrapper.AcquireCredentialsHandle(ISSPIInterface secModule, String package, CredentialUse intent, SCH_CREDENTIALS* scc) at
 System.Net.Security.SslStreamPal.AcquireCredentialsHandle(CredentialUse credUsage, SCH_CREDENTIALS* secureCredential) at 
System.Net.Security.SslStreamPal.AcquireCredentialsHandleSchCredentials(SslAuthenticationOptions authOptions) at 
System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions, Boolean newCredentialsRequested) --- End of inner exception stack trace --- at 
System.Net.Security.SslStreamPal.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions, Boolean newCredentialsRequested) at
 System.Net.Security.SslStream.AcquireCredentialsHandle(SslAuthenticationOptions sslAuthenticationOptions, Boolean newCredentialsRequested) at
 System.Net.Security.SslStream.AcquireClientCredentials(Byte[]& thumbPrint, Boolean newCredentialsRequested) at 
System.Net.Security.SslStream.GenerateToken(ReadOnlySpan`1 inputBuffer, Int32& consumed) at 
System.Net.Security.SslStream.NextMessage(ReadOnlySpan`1 incomingBuffer, Int32& consumed) at 
System.Net.Security.SslStream.ProcessTlsFrame(Int32 frameSize) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at 
System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at 
System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at 
System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at 
System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem) at 
System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) at 
System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at
 System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at 
System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at 
System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at 
NewAPI.WebClients.ServiceClient.GetToken()

I added this to see if I could have any insight on the ssl errors, but the breakpoint was never reached in .Net 9 RC2, while it was on .Net 8

 ServerCertificateCustomValidationCallback = (a, b, c, d) =>
 {
     return true;
 }

[CarnaViire]

I tested both proposed workarounds, singleton and adding loader limits with no success.

Then it confirms it's not HttpClientFactory, but HttpClient / HttpMessageHandler certificate issue...

I added this to see if I could have any insight on the ssl errors, but the breakpoint was never reached in .Net 9 RC2, while it was on .Net 8

ServerCertificateCustomValidationCallback is a different part of the handshake, it's validating the cert received from the server, while the problem you described was about a cert not being sent from the client...

I guess you can try using SocketsHttpHandler instead of HttpClientHandler, it has more configuration options available in SocketsHttpHandler.SslOptions, e.g. there you can specify a client cert selection callback to check whether it's called or not ( SocketsHttpHandler.SslOptions.LocalCertificateSelectionCallback)

[CarnaViire]

Quick search for the error "The credentials supplied to the package were not recognized" suggests that it is usually related to the application lacking permissions to access the cert's private key.... @rzikm do you know if anything was changed around that part?

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones See info in area-owners.md if you want to be subscribed.

[wfurt]

Looking at the second capture picture: the connection is closed by the client even before server responded. Normally the certificate is sent only after server asks for it. From the 0x8009030D it feels like the key is not accessible to underlying schannel. Perhaps @bartonjs can speculate on what changed and how to debug it.

[eeaquino]

Confirming it is access related, Ran VS as administrator and the certificate was sent as expected.