ServerCertificateCustomValidationCallback not called when client certificate used and server uses self sign certificate

[tymtam2]

Description

I am unable to establish a successful connection to a server which presents a self-signed certificate when using a client certificate with HttpClientHandler and HttpClient. (Please see the code in the Reproduction Steps section). ServerCertificateCustomValidationCallback is never called.

I can successfully connect to the server using the same pfx file with:

• curl --insecure --cert-type P12 --cert xxx.pfx:XXX -X GET https://test.example.com:443/aaa
• Postman
• a Rust program

Reproduction Steps

var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
handler.ServerCertificateCustomValidationCallback = (HttpRequestMessage message, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors)
    => { return CheckCustomServerCert(log, message, certificate, chain, sslPolicyErrors); };

// File is output of "openssl pkcs12 -export -out withchain.pfx -inkey priv_key_client.pem -in client.crt -certfile chain.crt"
var filePath = "...withchain.pfx";
handler.ClientCertificates.Add(new X509Certificate2(fileName: filePath, password: "..."));

var client = new HttpClient(handler);
var result = await client.GetAsync("...").ConfigureAwait(false); // throws here

Expected behavior

Objective 1: ServerCertificateCustomValidationCallback is called Objective 2: Connection can be established

Actual behavior

Windows 10 and 11:

System.Net.Http: The SSL connection could not be established, see inner exception. System.Net.Security: Authentication failed, see inner exception. The certificate chain was issued by an authority that is not trusted.

Linux (e.g. Ubuntu 18.04.1 LTS)

System.Net.Http: The SSL connection could not be established, see inner exception. System.Net.Security: Authentication failed, see inner exception. System.Net.Security: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca.

Regression?

Unsure

Known Workarounds

Not known

Configuration

.net6.0 Linux and Windows, x86 and x64

Other information

Similar but different? https://github.com/dotnet/runtime/issues/20671

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description I am unable to establish a successful connection to a server which presents a self-signed certificate when using a client certificate with `HttpClientHandler` and `HttpClient`. (Please see the code in the *Reproduction Steps* section). `ServerCertificateCustomValidationCallback ` is never called. I can successfully connect to the server using the same pfx file with: * `curl --insecure --cert-type P12 --cert xxx.pfx:XXX -X GET https://test.example.com:443/aaa` * Postman * a Rust program ### Reproduction Steps ``` try { var handler = new HttpClientHandler(); handler.ClientCertificateOptions = ClientCertificateOption.Manual; handler.SslProtocols = SslProtocols.Tls12; handler.ServerCertificateCustomValidationCallback = (HttpRequestMessage message, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors) => { return CheckCustomServerCert(log, message, certificate, chain, sslPolicyErrors); }; // File is output of "openssl pkcs12 -export -out withchain.pfx -inkey priv_key_client.pem -in client.crt -certfile chain.crt" var filePath = "...withchain.pfx"; if (!File.Exists(filePath)) { throw new Exception($"Certificate not found at '{filePath}'"); } handler.ClientCertificates.Add(new X509Certificate2(fileName: filePath, password: "...")); var client = new HttpClient(handler); // TODO have a static one var result = await client.GetAsync("...").ConfigureAwait(false); // throws here var content = await result.Content.ReadAsStringAsync().ConfigureAwait(false); log.LogInformation(content); } catch (Exception ex) { log.LogWarning(ex, "Sad"); } (...) static bool CheckCustomServerCert(ILogger log, HttpRequestMessage message, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors) { log.LogInformation("Checking server cert"); return true; } ``` ### Expected behavior Objective 1: ServerCertificateCustomValidationCallback is called Objective 2: Connection can be established ### Actual behavior Windows 10 and 11: `System.Net.Http: The SSL connection could not be established, see inner exception. System.Net.Security: Authentication failed, see inner exception. The certificate chain was issued by an authority that is not trusted.` Linux (e.g. Ubuntu 18.04.1 LTS) `System.Net.Http: The SSL connection could not be established, see inner exception. System.Net.Security: Authentication failed, see inner exception. System.Net.Security: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca.` ### Regression? Unsure ### Known Workarounds Not known ### Configuration .net6.0 Linux and Windows, x86 and x64 ### Other information Similar but different? https://github.com/dotnet/runtime/issues/20671

Author:	tymtam2
Assignees:	-
Labels:	`area-System.Net.Security`, `untriaged`
Milestone:	-

[rzikm]

I am afraid I can't reproduce this behavior:

using System.Net.Security;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;

var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
handler.ServerCertificateCustomValidationCallback = (HttpRequestMessage message, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors)
    =>
{
    System.Console.WriteLine("Got here");
    return true;
};

handler.ClientCertificates.Add(new X509Certificate2(fileName: @"C:\source\DotnetSandbox\contoso.com.pfx", password: "testcertificate"));

var client = new HttpClient(handler);
var result = await client.GetAsync("https://self-signed.badssl.com/").ConfigureAwait(false); // no exception here

the client certificate should not even be considered unless server explicitly asks for client certificate (it is not sent by default by clients).

Can you perhaps create a self-contained repro (a small project) which we can download and run locally to reproduce the issue?

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[wfurt]

can you do packet capture @tymtam2? ssl3_read_bytes:tlsv1 alert unknown ca. smells like server no liking you client certificate. e.g. this is not client problem. It can also be because the server needs intermediate certificates from the pfx. You would see that as well if you compare it with curl.

[tymtam2]

@wfurt > smells like server no liking you client certificate. e.g. this is not client problem.

I'd say that in this case it would not work with the same file from Rust, Postman, curl and with openssl s_client?

Different errors in linux and Windows strongly suggest for me it's a client issue, not the server's.

I'll keep investigating and report back.

[wfurt]

looks what is in that pfx. You can always create standalone repo...

[tymtam2]

rzikm,

Thanks for trying to reproduce this. I think https://self-signed.badssl.com is not a good test case because this server doesn't require a client cert so the handshake is different. I'm not that comfortable in this area but if the client cert is not sent by default - which you point out - then using https://self-signed.badssl.com won't use the same path.

I wonder if you have a comment on why would the same pfx file work with curl, Rust, Postman and also with openssl s_client? With these I can connect successfully using the same file [the same file on the hard drive to eliminate any file issues)

For "raw" openssl the successful connection it achieved using:

openssl s_client -connect serverX.com.au:443 -cert client.crt -key priv_key_client.pem -CAfile client_chain.crt

[wfurt]

Does it work for you @tymtam2 if you skip client_chain.crt? You certainly don't do that pat with your c# snippet.

[tymtam2]

can you do packet capture @tymtam2?

I'll look into that as well, but is there something I could do to get more logs from httpClient(Handler) to diagnose this?

https://github.com/dotnet/runtime/issues/20671 says that

"starting with .NET Core 2.1 Preview 2, the default HTTP implementation uses the new SocketsHttpHandler (not WinHTTP)"

and this feels me with hope of being able of getting more logs (at least on Windows) :).

[wfurt]

Yes, you can get more logs. But typically they don't cover details of the handshake. I feel packet captures and test without the chain would be most valuable for triage.

[tymtam2]

test without the chain

What is "test without the chain"?

[wfurt]

From the above

openssl s_client -connect serverX.com.au:443 -cert client.crt -key priv_key_client.pem -CAfile client_chain.crt

change it to

openssl s_client -connect serverX.com.au:443 -cert client.crt -key priv_key_client.pem

[tymtam2]

Does it work for you @tymtam2 if you skip client_chain.crt? You certainly don't do that pat with your c# snippet.

It's part of the snippet via:
// File is output of "openssl pkcs12 -export -out withchain.pfx -inkey priv_key_client.pem -in client.crt -certfile chain.crt

[tymtam2]

With openssl s_client -connect serverX.com.au:443 -cert client.crt -key priv_key_client.pem the it results in:

Verify return code: 20 (unable to get local issuer certificate)

[tymtam2]

Does it work for you @tymtam2 if you skip client_chain.crt? You certainly don't do that pat with your c# snippet.

It's part of the snippet via: // File is output of "openssl pkcs12 -export -out withchain.pfx -inkey priv_key_client.pem -in client.crt -certfile chain.crt

I wonder if you're saying that the chain cert needs to be supplied in a different way with HttpClientHandler and one pfx is not enough (I know I keep banging about it but this one pfx - with 3 certs [client, and two in the chain] and the private key - is enough when using Rust, postman and curl).

[wfurt]

yes. But I wanted some evidence before making the claim. Server may be ok without the intermediates so seeing the exchange for both cases is useful. So as test with other clients where the software does not have access to the intermediates.

Basically you would need to do

X509Certificate2Collection certs = new X509Certificate2Collection("withchain.pfx");

and add that to User's "Ca" store e.g. places where intermediates are stored. There is currently no good way to pass them in directly.

[tymtam2]

After adding the crt with the CA cert chain it works:

1. ServerCertificateCustomValidationCallback is called
2. the connection is successful.

After this spectacular success, I also checked if using a client.pfx that doesn't have the ca chain works, and it does.

A working snippet would be something like this:

var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
handler.ServerCertificateCustomValidationCallback += (HttpRequestMessage message, X509Certificate2? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors)
    =>
    {
        System.Console.WriteLine("Checking server cert");;
        return true;
    };

// Change 1: (Optional) Not using ca_chain.crt so this may be either of:  
// openssl pkcs12 -export -out client.pfx -inkey priv_key_client.pem -in client.crt -certfile ca_chain.crt
// openssl pkcs12 -export -out client.pfx -inkey priv_key_client.pem -in client.crt 
var clientCertFileName = "client.pfx"; //pfx
var clientCertPKandChain = new X509Certificate2(fileName: folder + clientCertFileName, password: passphrase);
handler.ClientCertificates.Add(clientCertPKandChain);

// Change 2: register ca chain into the store: 
var caChainCertFileName = "ca_chain.crt"; //crt
var caChainCert = new X509Certificate2(fileName: folder + caChainCertFileName);
var store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadWrite);
store.Add(caChainCert);

var client = new HttpClient(handler); 
var result = await client.GetAsync("https://serverX.com.au:443/dcap").ConfigureAwait(false); // Success

Observations:

1. Having to add the server's CA chain to the store makes ServerCertificateCustomValidationCallback seem not that powerful ;)
2. Would the X509Stores (current user's or local machine's) be writeable from an Azure Function? (What if there many running in parallel? How to manage adding the cert?) (I will of course check this, but a comment would be appreciated)

[wfurt]

This will eventually improve with https://github.com/dotnet/runtime/issues/71194 but I generally agree with the observation. I think it would still be interesting to see the packet captures with original issue but I understand that you may not want to since you have working solution.

Linux really does not have concept of certificate stores (unlike macOS and Windows). All the user stores are emulated on top of files in home directory - but I don't know if that is available from AF. Adding in parallel should be safe AFAIK. The certificates are indexed by hash and as far as I know adding existing certificate does not create another copy.

[wfurt]

BTW You can also probably fix it with adding the intermediates to the server ... if you have control over it. RFC says client should send the intermediates but the TLS does not really depend on it.

[tymtam2]

I somewhat have control (via email to another organisation) and this may be the way to address this.

In my somewhat rudimentary understanding of TLS (1.2) the intermediates should not play a role. However, in work not related to what's happening here we've discovered that we need to supply the intermediate certs when connecting to Azure Device Provisioning Service even though the intermediate certs are uploaded to the DPS via a separate channel. We've just learned to live with this.

Apropos packet capture, what are you after? I should be able to set it up after the weekend. I have control only over the client so the data would be only from the client's perspective. Would this be still of interest?

[wfurt]

Yes, client side will be sufficient. My suspicion is that the server is failing validate client's certificate without the intermediates and therefore aborting the session. We should see that when looking at the TLS exchange.

[tymtam2]

This log is from Windows and results in System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> System.ComponentModel.Win32Exception (0x80090325): The certificate chain was issued by an authority that is not trusted.

"No.","Time","Source","Destination","Protocol","Length","Info"
"22149","1090.883660","192.168.1.23","se.rv.er","TCP","66","56474  >  443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1"
"22150","1090.902054","se.rv.er","192.168.1.23","TCP","66","443  >  56474 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128"
"22151","1090.902172","192.168.1.23","se.rv.er","TCP","54","56474  >  443 [ACK] Seq=1 Ack=1 Win=131328 Len=0"
"22152","1090.953562","192.168.1.23","se.rv.er","TLSv1.2","245","Client Hello"
"22153","1090.972912","se.rv.er","192.168.1.23","TLSv1.2","1514","Server Hello"
"22154","1090.972912","se.rv.er","192.168.1.23","TLSv1.2","487","Certificate, Server Key Exchange, Certificate Request, Server Hello Done"
"22155","1090.973017","192.168.1.23","se.rv.er","TCP","54","56474  >  443 [ACK] Seq=192 Ack=1894 Win=131328 Len=0"
"22156","1090.987254","192.168.1.23","se.rv.er","TLSv1.2","869","Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message"
"22157","1091.005783","se.rv.er","192.168.1.23","TLSv1.2","61","Alert (Level: Fatal, Description: Unknown CA)"
"22158","1091.005783","se.rv.er","192.168.1.23","TCP","54","443  >  56474 [FIN, ACK] Seq=1901 Ack=1007 Win=64128 Len=0"
"22159","1091.005898","192.168.1.23","se.rv.er","TCP","54","56474  >  443 [ACK] Seq=1007 Ack=1902 Win=131328 Len=0"
"22160","1091.009858","192.168.1.23","se.rv.er","TCP","54","56474  >  443 [FIN, ACK] Seq=1007 Ack=1902 Win=131328 Len=0"
"22161","1091.028144","se.rv.er","192.168.1.23","TCP","54","443  >  56474 [ACK] Seq=1902 Ack=1008 Win=64128 Len=0"

[wfurt]

right. so The client sends certificate (or not) and it gets "22157","1091.005783","se.rv.er","192.168.1.23","TLSv1.2","61","Alert (Level: Fatal, Description: Unknown CA)" from that server and that kills the session. That happens before client executes the validation logic of the server cert.

So the handshake fails because server fails to validate and kills the session.