search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.93k stars 4.64k forks source link

SSL RemoteCertificateNameMismatch on MacOS Catalina #666

Closed TingluoHuang closed 4 years ago

TingluoHuang commented 4 years ago

I am using Proxyman to decrypt https traffic from my self-contained netcore 3.0 console app on macOS. Like using Fiddler on Windows.

I am getting System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. with inner exception System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure when the proxy is on.

I added a customize ServerCertificateCustomValidationCallback to dump out the ssl error, request url and the certificate.

sslPolicyErrors: RemoteCertificateNameMismatch
HttpRequestUri: https://pipelines.actions.githubusercontent.com/WoxlUxJHrKEzIp4Nz3YmrmLlZBonrmj9xCJ1lrzcJ9ZsD1Tnw7/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1
Certificate:
[Version]
  V3

[Subject]
  OU=https://proxyman.io, CN=*.actions.githubusercontent.com, O="GitHub, Inc.", L=San Francisco, C=US
  Simple Name: *.actions.githubusercontent.com
  DNS Name: *.actions.githubusercontent.com

[Issuer]
  OU=https://proxyman.io, CN="Proxyman CA (1 Nov 2019, htl-mac.local)", O=Proxyman Ltd, L=Singapore, C=SG
  Simple Name: Proxyman CA (1 Nov 2019, htl-mac.local)
  DNS Name: Proxyman CA (1 Nov 2019, htl-mac.local)

[Serial Number]
  00EE6265BFC8F6A251

[Not Before]
  11/1/2019 2:15:21 PM

[Not After]
  2/3/2022 1:15:21 PM

[Thumbprint]
  D1657538605625A0D41B7195CCF80806682374DA

[Signature Algorithm]
  sha256RSA(1.2.840.113549.1.1.11)

[Public Key]
  Algorithm: RSA
  Length: 2048
  Key Blob: 3082010A0282010100E2788EC7C9FF6705B22295EB4782FAE49DC259B7A0C6C80A655BF6AC330E4449BEC11D2835AF4D2C855D39B0E0472A72500B38AA44D64F6250581DBCC937153306976B00EBAAEA864C08D5F4CAC13D445DF5FE2F18DC528FA01B8C82CA85899E1064DED6905A867032C38CE18B6B5E01870533C5B53F56F6D081D3FBADC2FC8DA5843FBC4D6E1146A7E0E2791802167EF09D890288E13876B8C0FDDE8FAF3719981745BB983C6393A569CF768146DB3F0E8264D5E0084EAD2471D6131E9C5FFC6DE0C53FE7566323746716809D4245C3B585220B9B3378418131BBE92DFAE8CFB511AF7CCEA12312A80F5BAB5F52FF39C76D4B5EA7BCAC6215DEDD7F935EB8D90203010001
  Parameters: 0500

[Extensions]
* X509v3 Key Usage(2.5.29.15):
  030204F0
* (2.5.29.17):
  DNS:*.actions.githubusercontent.com, DNS:actions.githubusercontent.com

The request url's authority seems match with the cert's CN. If i export DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 to force my app use the old curl http handler, the SSL error goes away. So I am not sure why I am getting SSL error on when use SocketHttpHandler.

davidsh commented 4 years ago

@wfurt

wfurt commented 4 years ago

Do you use environment variables or do you set proxy explicitly in via API? Short code fragment would be nice for clarity how to reproduce it.

wfurt commented 4 years ago

And do you have LANG=en_US.UTF-8 or do you have locale set to anything else?

TingluoHuang commented 4 years ago

@wfurt 

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

</Project>
using System;
using System.Net;
using System.Net.Http;
using System.Net.Security;

namespace proxytest
{
    public class WebProxy : IWebProxy
    {
        public ICredentials Credentials { get; set; }

        public Uri GetProxy(Uri destination)
        {
            return new Uri("http://127.0.0.1:9090");
        }

        public bool IsBypassed(Uri host)
        {
            return false;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");

            using (HttpClientHandler clientHandler = new HttpClientHandler())
            {
                clientHandler.Proxy = new WebProxy();
                clientHandler.ServerCertificateCustomValidationCallback = (sender, certificate, chain, sslPolicyErrors) =>
                {
                    if (sslPolicyErrors == SslPolicyErrors.None)
                    {
                        return true;
                    }
                    else
                    {
                        Console.WriteLine($"SSL ERROR:{sslPolicyErrors.ToString()}");
                        Console.WriteLine(sender.RequestUri.AbsoluteUri);
                        Console.WriteLine(certificate.ToString(true));

                        return false;
                    }
                };
                using (var httpClient = new HttpClient(clientHandler))
                {
                    var result = httpClient.GetStringAsync("https://github.com").GetAwaiter().GetResult();
                    Console.WriteLine(result);
                }
            }
        }
    }
}
ting@htl-mac proxytest % printenv|sort
HOME=/Users/ting
LANG=en_US.UTF-8
LOGNAME=ting
LaunchInstanceID=42DF0CC6-7E63-405A-B233-8A6E7C29719A
OLDPWD=/Users/ting/Desktop
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/share/dotnet:~/.dotnet/tools
PWD=/Users/ting/Desktop/proxytest
SECURITYSESSIONID=186a6
SHELL=/bin/zsh
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.CEawhAXvQp/Listeners
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=A8A7F4C6-E8A2-4C32-9954-A6D7295908A4
TMPDIR=/var/folders/h1/nkrscyxx6858j3wz7dvzcq5r0000gn/T/
USER=ting
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
_=/usr/bin/printenv
ting@htl-mac proxytest % dotnet run                                            
SSL ERROR:RemoteCertificateNameMismatch
https://github.com/
[Version]
  V3

[Subject]
  OU=https://proxyman.io, CN=github.com, O="GitHub, Inc.", L=San Francisco, C=US
  Simple Name: github.com
  DNS Name: github.com

[Issuer]
  OU=https://proxyman.io, CN="Proxyman CA (1 Nov 2019, htl-mac.local)", O=Proxyman Ltd, L=Singapore, C=SG
  Simple Name: Proxyman CA (1 Nov 2019, htl-mac.local)
  DNS Name: Proxyman CA (1 Nov 2019, htl-mac.local)

[Serial Number]
  00EE6265BFC8F6A254

[Not Before]
  12/8/2019 9:26:39 AM

[Not After]
  3/12/2022 9:26:39 AM

[Thumbprint]
  70396C36BA73798B784EA2941BF05BB230634EF7

[Signature Algorithm]
  sha256RSA(1.2.840.113549.1.1.11)

[Public Key]
  Algorithm: RSA
  Length: 2048
  Key Blob: 3082010A0282010100FD073FCE3DF3A256AF6AAD4DBDA5F21672355C386C408AC7FABE8AC1BA86D950DC414102CE24718C45F5971DDB62EEF5421D165B449832B3DF5A23626307FB258123114C8F96B367D7420FF4C90CDEFFBD42B0F4F3DE7D0B3123960F26DDD93407FC527360FDD25009B0277EDDB69DF3D5ACDE0706ABAA95E8E13C135F3B5137FD61762BC2CFCB319A96FD15E221BAED3C9610E078E8CEA89AE6627A5A455B78FE2CED65DFC32EC382BBBD6D75425BF59C7E3CE6D7503F2AFBE327C6A07B08EAD125CE318B7A8709ECC6CED2D79311794338E6A55F95E5F6CBAEEBFE801559655E40AD3ACD003E5FD7DD698BB2C01F386D287DF24C4143E94EDC34F52219CC110203010001
  Parameters: 0500

[Extensions]
* X509v3 Key Usage(2.5.29.15):
  030204F0
* (2.5.29.17):
  DNS:github.com, DNS:www.github.com

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
wfurt commented 4 years ago

I was not able to reproduce it @TingluoHuang I did run Wireshark to make sure request goes trough Proxman:

CONNECT github.com:443 HTTP/1.1
Host: github.com:443

HTTP/1.1 200 Connection Established

...........]...*...q..1..._.......B......S...,...,.+.$.#.
.   ...0./.(.'...........=.<.5./.
...L.....
..
github.com.
.................
.........................................Y...U..]....mf<%t5uuk.....x......<...[. {1..[..Q...[......U.......*&.R...0..
............................0...0.........  ..A...J.o0
.   *.H..
.....0..1.0 ..U....SG1.0...U... Singapore1.0...U.
..Proxyman Ltd1/0-..U...&Proxyman CA (9 Dec 2019, macik2.local)1.0...U....https://proxyman.io0..
191209194709Z.
220313194709Z0o1.0  ..U....US1.0...U...
San Francisco1.0...U.
..GitHub, Inc.1.0...U...
github.com1.0...U....https://proxyman.io0.."0
.   *.H..
..........0..

and I do see decrypted response details in Proxyman GUI. tested on 10.14.6

cc: @bartonjs in case he has some ideas.

bartonjs commented 4 years ago

We let the OS determine whether or not the hostname matches (https://github.com/dotnet/corefx/blob/release/3.0/src/Native/Unix/System.Security.Cryptography.Native.Apple/pal_ssl.c#L357-L462).

Maybe something else is going on that's making things return false, but essentially we and Safari should have the same answer.

wfurt commented 4 years ago

On that note, Proxyman changes OS setting so it should be easy to test.

TingluoHuang commented 4 years ago

I tried Safari, it works fine with Proxyman, SSL traffic in Safari get decrypted correctly. Any other information I can collect that would help investigation? BTW I am on 10.15.1

wfurt commented 4 years ago

I can reproduce it on Catalina. I will take a look.

wfurt commented 4 years ago

It looks like we get ksectrustresultrecoverabletrustfailure: https://developer.apple.com/documentation/security/sectrustresulttype/ksectrustresultrecoverabletrustfailure?language=objc The CA certificate does not have OSCP or CRL distribution list. It should not matter as we set revocation to NoCheck. But it seems like something changes at OS crypto layer. As @bartonjs mentioned, it is OS telling us not to trust that certificate. The name mismatch may be artifact of mapping OS errors.

TingluoHuang commented 4 years ago

Since Safari works fine with Proxyman, i assume there might be some different between Safari and Netcore ask OS to validate SSL cert, maybe some validation option difference.

wfurt commented 4 years ago

I'm still not sure what exactly is going on as Catalina sources are not available yet. Internally this fails with -2147408889 CSSMERR_APPLETP_INVALID_EXTENDED_KEY_USAGE (Invalid Extended Key Usage for policy). It should not matter as the certificate generated by Proxyman does not have any EKU.

vcsjones commented 4 years ago

@wfurt

I think perhaps Apple started requiring the EKU as per the CAB/F baseline requirements in 10.15. Apple has a support article on this here: https://support.apple.com/en-us/HT210176

Additionally, all TLS server certificates issued after July 1, 2019 (as indicated in the NotBefore field of the certificate) must follow these guidelines:

TLS server certificates must contain an ExtendedKeyUsage (EKU) extension containing the id-kp-serverAuth OID.

TingluoHuang commented 4 years ago

Proxyman fixed their CA cert and server certs when decrypt https traffic. This issue can be resolved now. Thanks @wfurt

bartonjs commented 4 years ago

Glad things are working for you now :smile:.

danmoseley commented 4 years ago

Was there more info we could have included in the exception that might have made this quicker to understand (like CSSMERR_APPLETP_INVALID_EXTENDED_KEY_USAGE)?