Connecting to server using SSL certificate in MacOS fails with code that success on Windows

[georgiosd]

Hey folks

I am using an SSL certificate to connect to RavenDB (doesn't seem like the issue is specific to RavenDB though).

The code is very simple as far as the certificate handling goes

var pfx = await keyvault.GetSecretAsync(...);
var bytes = Convert.FromBase64String(pfx);
var cert = new X509Certificate2(bytes);

This code works correctly on Windows, but fails on latest MacOS with the following exception - you can ignore the first couple of lines which are part of RavenDB's stack:

Unhandled exception. System.AggregateException: Failed to retrieve cluster topology from all known nodes
The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Internal.Cryptography.Pal.SecTrustChainPal.ParseResults(SafeX509ChainHandle chainHandle, X509RevocationMode revocationMode)
   at Internal.Cryptography.Pal.SecTrustChainPal.Execute(DateTime verificationTime, Boolean allowNetwork, OidCollection applicationPolicy, OidCollection certificatePolicy
, X509RevocationFlag revocationFlag)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolic
y, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
   at System.Net.SafeDeleteSslContext.SetCertificate(SafeSslHandle sslContext, X509Certificate2 certificate)
   at System.Net.SafeDeleteSslContext.CreateSslContext(SafeFreeSslCredentials credential, Boolean isServer)
   at System.Net.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, ArraySegment`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)

I am happy to help create a repro but it would require a secured RavenDB instance so may have to coordinate with someone looking into this.

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq Notify danmosemsft if you want to be subscribed.

[scalablecory]

@wfurt too

[vcsjones]

at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)

It looks like there is an exception being thrown while building a chain for the server certificate.

I assume this is being used as the server certificate from RavenDB:

var pfx = await keyvault.GetSecretAsync(...);
var bytes = Convert.FromBase64String(pfx);
var cert = new X509Certificate2(bytes);

What happens if you take the public certificate from there and try building a chain with it on macOS?

X509Chain chain = new X509Chain();
chain.Build(certificate);

Where certificate is the public part of the certificate from Azure Key Vault?

[vcsjones]

Looking further at the stack trace..

System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation. at Internal.Cryptography.Pal.SecTrustChainPal.ParseResults(SafeX509ChainHandle chainHandle, X509RevocationMode revocationMode)

It seems that the only CryptographicException thrown in ParseResults is https://github.com/dotnet/runtime/blob/85aee4d4914d17cd1f3b970405a9d1def82f18b8/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/ChainPal.cs#L297-L301

I suspect you are running in to https://github.com/dotnet/runtime/issues/32882, (or maybe another new chain status...). If my intuition is correct, then this is fixed in 3.1.4 and 2.1.18, but they are not yet released.

[georgiosd]

@vcsjones actually this is a client certificate, as I am trying to connect to the RavenDB instance. The client certificate is issued by RavenDB and is a derivative of the server certificate (while I don't understand the process).

Happy to wait for 3.1.4, is there an ETA? Not a pressing issue since it works on Windows.

[vcsjones]

@georgiosd ah, I see. Are you able to try .NET 5? I believe the fix for this is also in the latest preview. If you try on .NET 5 and change your app to target netcoreapp5.0, does it still reproduce?

If it still reproduces, then that means my hypothesis doesn't hold.

[georgiosd]

Sadly not fixed in 5 @vcsjones:

---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Internal.Cryptography.Pal.SecTrustChainPal.ParseResults(SafeX509ChainHandle chainHandle, X509RevocationMode revocationMode)
   at Internal.Cryptography.Pal.SecTrustChainPal.Execute(DateTime verificationTime, Boolean allowNetwork, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationFlag revocationFlag)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
   at System.Net.SafeDeleteSslContext.SetCertificate(SafeSslHandle sslContext, X509Certificate2 certificate)
   at System.Net.SafeDeleteSslContext.CreateSslContext(SafeFreeSslCredentials credential, Boolean isServer)
   at System.Net.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)

Building the chain does indeed fail, but the key has a private key in there too since it's a client cert.

Unhandled exception. System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at Internal.Cryptography.Pal.SecTrustChainPal.ParseResults(SafeX509ChainHandle chainHandle, X509RevocationMode revocationMode)
   at Internal.Cryptography.Pal.SecTrustChainPal.Execute(DateTime verificationTime, Boolean allowNetwork, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationFlag revocationFlag)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at ZeroToMvp.Nosolo.Program.Main(String[] args) in /Users/georgiosd/Documents/zerotomvp-nosolo/backend/Program.cs:line 33
   at ZeroToMvp.Nosolo.Program.<Main>(String[] args)

[vcsjones]

@georgiosd

Thanks. Does it still reproduce if you use the certificate without a private key? Something like:

X509Certificate2 certWithKey; //Your certificate
X509Certificate2 certWithoutKey = new X509Certificate2(certWithKey.Export(X509ContentType.Cert));

if (certWithoutKey.HasPrivateKey) {
    throw new InvalidOperationException();
}

X509Chain chain = new X509Chain();
chain.Build(certWithoutKey);

If that still throws, then it should be safe for you to share just the certificate without the private key. You can export certWithoutKey somewhere to disk using certWithoutKey.Export(X509ContentType.Cert). Once we have that, hopefully we can reproduce it.

[georgiosd]

@vcsjones indeed it does! Thanks for taking a look at this!

Here you go: https://pastebin.com/ttFeGiYN (expires in one day)

[vcsjones]

@georgiosd

Unfortunately this does not reproduce for me. X509Chain.Build returns false for me (as expected) but it doesn't throw.

On the Mac that reproduces this, can you save the certificate to disk (public should be fine), and run:

security verify-cert -v -C -c /path/to/your/certificate.cer

And share the complete output from that command?

The only output I get is "MissingIntermediate", which is to be expected since I don't have the intermediates on my machine.

---

Can you also share your macOS and .NET Core version you are using? I was unable to reproduce this on .NET Core 3.1.3 and macOS 10.15.4. Posting the output of dotnet --info would capture all of that.

[vcsjones]

Disregard the above. I was able to reproduce it. Thanks!

[vcsjones]

The issue here is that you have a leaf certificate that is pretending to be a CA.

However, the "client" certificate is issued from it, even though that issuing certificate has a Basic Constraints of CA:FALSE. So this is a misuse of using the certificate as an intermediate, when it isn't.

@bartonjs this results in a ChainStatus of "BasicConstraintsCA" and "BasicConstraintsPathLen", which we don't have a map for, hence the obscure CryptographicException.

I don't know why Windows is OK with using a certificate that has CA:FALSE as a CA.

[wfurt]

thanks @vcsjones for digging into this.

[georgiosd]

Thanks @vcsjones, I'm not sure what the resolution is here from RavenDB's POV so I will send them the issue so they can participate.

Will you please erase the reference to our domain and the link to the cert analysis? I think the rest of the description gives enough of a description. It's not particularly sensitive but still.

[vcsjones]

Will you please erase the reference to our domain and the link to the cert analysis? I think the rest of the description gives enough of a description. It's not particularly sensitive but still.

Sure. Removed from my post and deleted the edit history as well.

[georgiosd]

Thank you.

As a fellow dev, I'm curious though. How are you, as the dotnet core team, supposed to resolve this given that the experience seems to be different by OS design and the improper one seems to be Windows in this case?

[ayende]

One thing to note. We are intentionally doing this behavior (generating the cert from a server certificate). This is because we can't rely on having a CA: true certificate. Our certs are usually generated by Let's Encrypt.

We are fine with other software not respecting it, since we handle that internally inside RavenDB.

Note that across the board: Windows, Linux, Chrome, OpenSSL, etc - we have had no issue with this behavior.

[bartonjs]

This should be resolved by #35347.

[georgiosd]

Thank you @bartonjs @vcsjones - is this going to be in a 3.1.x release?

[bartonjs]

is this going to be in a 3.1.x release?

"probably"

We need to open a servicing request, and it needs to get approved. I don't expect pushback, but since I don't have the ability to actually say "yes", I'm hedging :smile:. I'm 99% sure it's too late for the May update, but June seems plausible.

[georgiosd]

@bartonjs I understand the hedging and approve 😆