search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.01k stars 4.67k forks source link

.NET Core 3.0 + OpenSSL 1.1: remote certificate is invalid according to the validation procedure #27362

Closed tmds closed 4 years ago

tmds commented 6 years ago

I'm trying the openssl1.1 support in .NET Core 3.0 and the sdk is unable to contact nuget.org:


Processing post-creation actions...
Running 'dotnet restore' on /root/console/console.csproj...
  Restoring packages for /root/console/console.csproj...
/root/dotnet/sdk/3.0.100-alpha1-009456/NuGet.targets(114,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/root/console/console.csproj]
/root/dotnet/sdk/3.0.100-alpha1-009456/NuGet.targets(114,5): error :   The SSL connection could not be established, see inner exception. [/root/console/console.csproj]
/root/dotnet/sdk/3.0.100-alpha1-009456/NuGet.targets(114,5): error :   The remote certificate is invalid according to the validation procedure. [/root/console/console.csproj]

Restore failed.
Post action failed.
Description: Restore NuGet packages required by this project.

Steps to reproduce

run a fedora:28 container:

$ docker run -ti fedora:28 /bin/bash

Inside the container, first install .NET Core:

dnf install libicu
curl -o dotnet.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-linux-x64.tar.gzcurl -o dotnet.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-linux-x64.tar.gz
mkdir dotnet
cd dotnet
tar xvf ../dotnet.tar.gz 
export PATH=`pwd`:$PATH
cd ..

Then try to create a project:

mkdir console
cd console
dotnet new console

The restore fails with the above error message.

Additinal info

Using a pre-compiled application inspecting the certificate chain shows this:

Chain Information
Chain revocation flag: ExcludeRoot
Chain revocation mode: Online
Chain verification flag: NoFlag
Chain verification time: 09/11/2018 15:52:44
Chain status length: 2
Chain application policy count: 1
Chain certificate policy count: 0 

Chain Element Information
Number of chain elements: 3
Chain elements synchronized? False 

Element issuer name: CN=Microsoft IT TLS CA 1, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Element certificate valid until: 02/23/2020 01:19:59
Element certificate is valid: False
Element error status length: 0
Element information: 
Number of element extensions: 10

Element issuer name: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE
Element certificate valid until: 05/20/2024 12:51:28
Element certificate is valid: False
Element error status length: 0
Element information: 
Number of element extensions: 8

Element issuer name: CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE
Element certificate valid until: 05/12/2025 23:59:00
Element certificate is valid: False
Element error status length: 2
Element information: 
Number of element extensions: 3

The ChainelementStatus[] for the last certificate contains: ExplicitDistrust and UntrustedRoot.

bartonjs commented 6 years ago

We're seeing the same thing while trying to stand up SLES 15 testing. Odd that Ubuntu 18.04 and Fedora 27 aren't saying this.

RheaAyase commented 6 years ago

Fedora packages are depending on compat-openssl10

bartonjs commented 6 years ago

@RheaAyase I don't think our test lab uses the packages. The portable shim is now hyperportable, and for .NET Core 3.0 it prefers libssl.so.1.1 over other targets. (If the package uses a crypto shim that was built as non-portable, then it'll be against whichever library/set of headers it was built)

In a debugger I see that for SLES15 and Ubuntu18.04 they both make it into obj_trust, and then take different paths at obj_trust+30

    0x7fff5d476330 <+0>:   pushq  %r15
    0x7fff5d476332 <+2>:   pushq  %r14
    0x7fff5d476334 <+4>:   movq   %rsi, %r14
    0x7fff5d476337 <+7>:   pushq  %r13
    0x7fff5d476339 <+9>:   pushq  %r12
    0x7fff5d47633b <+11>:  movl   %edx, %r13d
    0x7fff5d47633e <+14>:  pushq  %rbp
    0x7fff5d47633f <+15>:  pushq  %rbx
    0x7fff5d476340 <+16>:  subq   $0x8, %rsp
    0x7fff5d476344 <+20>:  movq   0x138(%rsi), %rbx
    0x7fff5d47634b <+27>:  testq  %rbx, %rbx
    0x7fff5d47634e <+30>:  je     0x7fff5d47637a            ; <+74>

Ubuntu jumps to +74, SLES continues to +32. So on Ubuntu there's no X509_CERT_AUX data, but on SLES there is. And I'm not sure where it's coming from.

SLES 15: [ 20] EC2150A5-B6A8-AC0C-DECC-CD1AE98108D2-5C750F17 /usr/lib64/libssl.so.1.1 [ 21] 9488A48E-4F58-0B29-9C00-D508C47289A0-3DE45E9A /usr/lib64/libcrypto.so.1.1 (lldb) p/x (uint32_t)OpenSSL_version_num() (uint32_t) $2 = 0x1010008f

$ openssl version -a OpenSSL 1.1.0h-fips 27 Mar 2018 built on: reproducible build, date unspecified platform: linux-x86_64 options: bn(64,64) rc4(16x,int) des(int) blowfish(ptr) compiler: gcc -DZLIB -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -D_FORTIFY_SOURCE=2 -DTERMIO -DPURIFY -D_GNU_SOURCE -DOPENSSL_NO_BUF_FREELISTS -DOPENSSLDIR="\"/etc/ssl\"" -DENGINESDIR="\"/usr/lib64/engines-1.1\"" -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -std=gnu99 -Wa,--noexecstack -fno-common -Wall -Wa,--noexecstack OPENSSLDIR: "/etc/ssl" ENGINESDIR: "/usr/lib64/engines-1.1"

Ubuntu 18.04 [ 25] 9787FF1F-FFE8-D822-68D8-9F6BFA3AF07B-3E9D140F /usr/lib/x86_64-linux-gnu/libssl.so.1.1 [ 26] DA6345B4-1E06-A900-1B5C-51DEAF0784BE-9D49810D /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (lldb) p/x (uint32_t)OpenSSL_version_num() (uint32_t) $1 = 0x1010007f

$ openssl version -a OpenSSL 1.1.0g 2 Nov 2017 built on: reproducible build, date unspecified platform: debian-amd64 compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/x86_64-linux-gnu/engines-1.1\"" OPENSSLDIR: "/usr/lib/ssl" ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"

omajid commented 6 years ago

Apologies in advance for possibly not helpful advice:

If you enable the debug repositories on SLES (via Yast, for example), you should be able to debuginfo-install openssl (or, better yet, debuginfo-install $name-and-version-of-openssl-package) to pull down the exact matching sources used to compile the binary. Hopefully that makes it easier to debug this than looking at the disassembly.

bartonjs commented 6 years ago

@omajid Very helpful, actually :smile:.

Looks like the ca-certificates package took a bit of an update:

Ubuntu 18.04

$ cat Baltimore_CyberTrust_Root.pem
-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
-----END CERTIFICATE-----

SLES 15

$ cat Baltimore_CyberTrust_Root.pem
-----BEGIN TRUSTED CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmpMDEwFAYIKwYBBQUHAwQGCCsGAQUF
BwMBDBlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290
-----END TRUSTED CERTIFICATE-----

$ openssl x509 -in Baltimore_CyberTrust_Root.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 33554617 (0x20000b9)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
        Validity
            Not Before: May 12 18:46:00 2000 GMT
            Not After : May 12 23:59:00 2025 GMT
        Subject: C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a3:04:bb:22:ab:98:3d:57:e8:26:72:9a:b5:79:
                    d4:29:e2:e1:e8:95:80:b1:b0:e3:5b:8e:2b:29:9a:
                    64:df:a1:5d:ed:b0:09:05:6d:db:28:2e:ce:62:a2:
                    62:fe:b4:88:da:12:eb:38:eb:21:9d:c0:41:2b:01:
                    52:7b:88:77:d3:1c:8f:c7:ba:b9:88:b5:6a:09:e7:
                    73:e8:11:40:a7:d1:cc:ca:62:8d:2d:e5:8f:0b:a6:
                    50:d2:a8:50:c3:28:ea:f5:ab:25:87:8a:9a:96:1c:
                    a9:67:b8:3f:0c:d5:f7:f9:52:13:2f:c2:1b:d5:70:
                    70:f0:8f:c0:12:ca:06:cb:9a:e1:d9:ca:33:7a:77:
                    d6:f8:ec:b9:f1:68:44:42:48:13:d2:c0:c2:a4:ae:
                    5e:60:fe:b6:a6:05:fc:b4:dd:07:59:02:d4:59:18:
                    98:63:f5:a5:63:e0:90:0c:7d:5d:b2:06:7a:f3:85:
                    ea:eb:d4:03:ae:5e:84:3e:5f:ff:15:ed:69:bc:f9:
                    39:36:72:75:cf:77:52:4d:f3:c9:90:2c:b9:3d:e5:
                    c9:23:53:3f:1f:24:98:21:5c:07:99:29:bd:c6:3a:
                    ec:e7:6e:86:3a:6b:97:74:63:33:bd:68:18:31:f0:
                    78:8d:76:bf:fc:9e:8e:5d:2a:86:a7:4d:90:dc:27:
                    1a:39
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                E5:9D:59:30:82:47:58:CC:AC:FA:08:54:36:86:7B:3A:B5:04:4D:F0
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:3
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
    Signature Algorithm: sha1WithRSAEncryption
         85:0c:5d:8e:e4:6f:51:68:42:05:a0:dd:bb:4f:27:25:84:03:
         bd:f7:64:fd:2d:d7:30:e3:a4:10:17:eb:da:29:29:b6:79:3f:
         76:f6:19:13:23:b8:10:0a:f9:58:a4:d4:61:70:bd:04:61:6a:
         12:8a:17:d5:0a:bd:c5:bc:30:7c:d6:e9:0c:25:8d:86:40:4f:
         ec:cc:a3:7e:38:c6:37:11:4f:ed:dd:68:31:8e:4c:d2:b3:01:
         74:ee:be:75:5e:07:48:1a:7f:70:ff:16:5c:84:c0:79:85:b8:
         05:fd:7f:be:65:11:a3:0f:c0:02:b4:f8:52:37:39:04:d5:a9:
         31:7a:18:bf:a0:2a:f4:12:99:f7:a3:45:82:e3:3c:5e:f5:9d:
         9e:b5:c8:9e:7c:2e:c8:a4:9e:4e:08:14:4b:6d:fd:70:6d:6b:
         1a:63:bd:64:e6:1f:b7:ce:f0:f2:9f:2e:bb:1b:b7:f2:50:88:
         73:92:c2:e2:e3:16:8d:9a:32:02:ab:8e:18:dd:e9:10:11:ee:
         7e:35:ab:90:af:3e:30:94:7a:d0:33:3d:a7:65:0f:f5:fc:8e:
         9e:62:cf:47:44:2c:01:5d:bb:1d:b5:32:d2:47:d2:38:2e:d0:
         fe:81:dc:32:6a:1e:b5:ee:3c:d5:fc:e7:81:1d:19:c3:24:42:
         ea:63:39:a9
Trusted Uses:
  E-mail Protection, TLS Web Server Authentication
No Rejected Uses.
Alias: Baltimore CyberTrust Root
-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
-----END CERTIFICATE-----

So the platform has said that the Baltimore root is only valid for Email and TlsServer. When we ask for a chain we end up on a path that asks X509_check_trust for X509_TRUST_DEFAULT, which asks for trust for NID_anyExtendedKeyUsage. OpenSSL 1.1's validator says that if there were any explicit yesses then they're restricted to that, and since "any" isn't in { Email, TlsServer } it is prohibited, therefore the certificate is disallowed.

All of this seems to be because we're reading the system trust via PEM_read_bio_X509_AUX, which must not be what the OpenSSL commandline is doing (at least in 1.1).

tmds commented 6 years ago

X509_CERT_AUX data, but on SLES there is. And I'm not sure where it's coming from.

There are two bundle files on Fedora, a tls-ca-bundle.pem and a ca-bundle.trust.crt. Both have a # Baltimore CyberTrust Root certificate:

# Baltimore CyberTrust Root
-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp
-----END CERTIFICATE-----

and

# Baltimore CyberTrust Root
-----BEGIN TRUSTED CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX
DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y
ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy
VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr
mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr
IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK
mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu
XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy
dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye
jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1
BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3
DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92
9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx
jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0
Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz
ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS
R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmpMDEwFAYIKwYBBQUHAwQGCCsGAQUF
BwMBDBlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290
-----END TRUSTED CERTIFICATE-----

Notice this last one (from ca-bundle.trust.crt) starts with the same data with some additional stuff at the end, this is probably the AUX data.

So the platform has said that the Baltimore root is only valid for Email and TlsServer. When we ask for a chain we end up on a path that asks X509_check_trust for X509_TRUST_DEFAULT, which asks for trust for NID_anyExtendedKeyUsage. OpenSSL 1.1's validator says that if there were any explicit yesses then they're restricted to that, and since "any" isn't in { Email, TlsServer } it is prohibited, therefore the certificate is disallowed. All of this seems to be because we're reading the system trust via PEM_read_bio_X509_AUX, which must not be what the OpenSSL commandline is doing (at least in 1.1).

Sounds like the root cause! :tada:

filipnavara commented 6 years ago

@tmds Great find, the additional data are the following:

SEQUENCE (2 elem)
  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 1.3.6.1.5.5.7.3.4 emailProtection (PKIX key purpose)
    OBJECT IDENTIFIER 1.3.6.1.5.5.7.3.1 serverAuth (PKIX key purpose)
  UTF8String Baltimore CyberTrust Root

The ASN definition from OpenSSL is

ASN1_SEQUENCE(X509_CERT_AUX) = {
        ASN1_SEQUENCE_OF_OPT(X509_CERT_AUX, trust, ASN1_OBJECT),
        ASN1_IMP_SEQUENCE_OF_OPT(X509_CERT_AUX, reject, ASN1_OBJECT, 0),
        ASN1_OPT(X509_CERT_AUX, alias, ASN1_UTF8STRING),
        ASN1_OPT(X509_CERT_AUX, keyid, ASN1_OCTET_STRING),
        ASN1_IMP_SEQUENCE_OF_OPT(X509_CERT_AUX, other, X509_ALGOR, 1)
} ASN1_SEQUENCE_END(X509_CERT_AUX)

That translates to ->trust containing emailProtection and serverAuth and ->alias containing Baltimore CyberTrust Root.

tmds commented 6 years ago

@bartonjs is the fix to change from PEM_read_bio_X509_AUX to PEM_read_bio_X509? I gave it a try and it is working fine: all tests passed and I can now access nuget.org.

filipnavara commented 6 years ago

Hmm, I wondered about the same thing, but I guess depending on the code path it may not be the correct fix. It's possible to change the trust verification parameters to accept any extended key usage, or just some specific ones. It requires an explicit call to do that though (and not a very obvious one!) and I am not sure whether the expected verification options are available at the particular code location.

bartonjs commented 6 years ago

Wow, it's... way more subtle than that. It's that I read in a different order than the commandline.

CLI Defaults

$ openssl verify -verbose -show_chain microsoft-com-issuer.cer
microsoft-com-issuer.cer: OK
Chain:
depth=0: C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 EV SSL CA - G3 (untrusted)
depth=1: C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5

no-CApath (don't use the "each file in this directory" loader)

$ openssl verify -verbose -show_chain -no-CApath microsoft-com-issuer.cer
microsoft-com-issuer.cer: OK
Chain:
depth=0: C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 EV SSL CA - G3 (untrusted)
depth=1: C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5

no-CAfile (don't use the "all the certs are in this one big file" loader)

$ openssl verify -verbose -show_chain -no-CAfile microsoft-com-issuer.cer
C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
error 28 at 1 depth lookup: certificate rejected
error microsoft-com-issuer.cer: verification failed

What's error 28?

# define         X509_V_ERR_CERT_REJECTED                        28

Bingo.

Looks like putting the SSL_CERT_FILE contents before SSL_CERT_DIR contents will solve this particular problem. And then it matches the openssl CLI, which is my normal heuristic.

tmds commented 6 years ago

Looks like putting the SSL_CERT_FILE contents before SSL_CERT_DIR contents will solve this particular problem. And then it matches the openssl CLI, which is my normal heuristic.

@bartonjs The fix doesn't take into account that the issue comes from the AUX data in the certificates.

For example on Fedora 28 defaults are:

SSL_CERT_FILE=/etc/pki/tls/cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem  (no AUX)
SSL_CERT_DIR=/etc/pki/tls/certs

The SSL_CERT_DIR contains:
ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem                        (no AUX, same file as above)
ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt            (AUX)

By swapping the order, the no AUX certs are used.

Now consider when I have a use-case where I use override SSL_CERT_FILE and point it to a certificate bundle (e.g. containing a certificate for the trusting the corporate proxy server). Here, I can get back into the situation where the AUX data causes the certificates to be discarded.

I think a fix involves ignoring the AUX data (e.g. using PEM_read_bio_X509 instead of PEM_read_bio_X509_AUX) Or working with more specific trust than X509_TRUST_DEFAULT.

CC @krwq @wfurt

bartonjs commented 6 years ago

Yeah, I know that there's still a potential issue left. Flipping the order makes two things be true:

  1. We give the same answer as openssl verify
  2. The tests pass

If a restricted trust certificate is put in SSL_CERT_FILE (and matters) then two things would happen

  1. openssl verify would say no a. And presumably a lot of things (us included) would start to fail.
  2. We'd have to figure out what to do instead.

I'm assuming (perhaps naively) that whomever was making the change would test it with openssl verify. I'm also assuming that if it was something bigger (like changing the ca-certificates package or extractor) that a lot of other things would fail, and then the change would get rejected.

Since OpenSSL internally uses PEM_read_bio_X509_AUX it vaguely feels like it's more correct to let the data load, even if it has consequences. Should we decide that we don't actually want that, there are three potential changes (off the top of my head)

  1. PEM_read_bio_X509_AUX => PEM_read_bio_X509 (but, presumably, this fails to load the "TRUSTED" format, which could lead to a different type of problem)
  2. Clear the AUX data after loading (just blanket ignore it)
  3. Figure out some other trust value to use while chain building

Matching the openssl verify command (when using OpenSSL) puts us in a place of predictability and sets things up in a manner where they can be discussed. What I'm mainly missing on knowing what the next step is is why the ca-certificates bundle started changing, and what they expect to happen.

filipnavara commented 6 years ago

PEM_read_bio_X509_AUX => PEM_read_bio_X509 (but, presumably, this fails to load the "TRUSTED" format, which could lead to a different type of problem)

It doesn't fail. The AUX data is just appended past the normal X509 data and it is ignored and not loaded.

Clear the AUX data after loading (just blanket ignore it)

Not necessary since that's same as using PEM_read_bio_X509 minus error checking of the ignored data.

bartonjs commented 6 years ago

PEM_read_bio_X509_AUX => PEM_read_bio_X509 (but, presumably, this fails to load the "TRUSTED" format, which could lead to a different type of problem)

It doesn't fail.

That doesn't match the test I just did. If I copy the "BEGIN TRUSTED CERTIFICATE" form from above into a file and use a program that just reads and prints that cert, it fails to read it if I change the shim (Ubuntu 16.04 / OpenSSL 1.0.2).

PEM_read_bio_X509 only seeks for "BEGIN CERTIFICATE". PEM_read_bio_X509_AUX looks for either "BEGIN CERTIFICATE" or "BEGIN TRUSTED CERTIFICATE". So changing from X509_AUX to X509 would mean that "BEGIN TRUSTED CERTIFICATE" entries count as not being present. (okay, either of them also accept "BEGIN X509 CERTIFICATE", as a legacy compat thing)

filipnavara commented 6 years ago

Seems like you are right and I made two incorrect assumptions. I have already disposed of the virtual machine that I used for testing, so it would take me a while to reproduce the experiment. I've now consulted the OpenSSL 1.1.1 source code and it behaves as you say.

The mistake I did in my test is that I kept "BEGIN CERTIFICATE" in my test data and just replaced the base64 content. It then simply proceeds to d2i_X509 instead of d2i_X509_AUX and ignores the extra data. In fact d2i_X509_AUX calls d2i_X509 internally and expects that behavior and there's no further check of whether it consumed all the data.

The second mistake I made is that I interpreted this comment wrong:

is the fix to change from PEM_read_bio_X509_AUX to PEM_read_bio_X509? I gave it a try and it is working fine: all tests passed and I can now access nuget.org.

It very likely only worked because it skipped over the TRUSTED PEM files as invalid and got further to the non-trusted ones.

tmds commented 6 years ago

I think we should either:

These are options 1 and 3 from https://github.com/dotnet/corefx/issues/32224#issuecomment-421359576.

On Fedora certificates are available in both formats. Probably it is similar on other distros. Since option 1 is the simplest maybe we should start with that. If necessary, we can look at option 3.

bartonjs commented 6 years ago

So while I'm not sure why Fedora 28 (and one claim of Ubuntu 18.04) have hit this, too; but on SLES it's because of https://build.opensuse.org/package/view_file/security:tls/openssl-1_1/openssl-truststore.patch?expand=1.

While /etc/ssl/certs still uses the "BEGIN CERTIFICATE" form (and that's where the output of openssl version -a would send you), they changed X509_CERT_DIR and X509_CERT_FILE to not be based off of OPENSSLDIR. So it really is that SLES has put the default trust list into an interesting state; but probably don't notice much because the X509_CERT_FILE copy of the cert wins under default chain engines.

tmds commented 6 years ago

@bartonjs wdyt about the options mentioned earlier?

bartonjs commented 6 years ago

@tmds Based on the fact that this is a patch into OpenSSL by SLES to put OpenSSL itself into this ticking timebomb state, I think I want to leave things as they are in master. If you can help provide understanding as to why things are the way that they are, then I'd likely have a different position. My main concern is that if someone is using a BEGIN TRUSTED CERTIFICATE now to register deny reasons (and happens to deny id-eku-anyPurpose) then we're (correctly) reporting the chain as untrustworthy. Technically I could have already broken this state with the "which copy wins" change, but if there's only one copy either other change would be a breaking one for them.

I looked into alternate trust values, and they all seem differently worse; the most plausible would be TLS-server, since that seems to be what criterion the multi-PEM is built from (on SLES, at least).

tmds commented 5 years ago

I discussed this with Tomáš Mráz (@t8m) who maintains the OpenSSL packages for RHEL and Fedora. On short term it should be ok to use the certificates without trust data, on long term we should look at using the trust data and match the appropriate trust.

We can change from PEM_read_bio_X509_AUX to PEM_read_bio_X509 where we read the store certificates to use the certificates without trust data.

filipnavara commented 5 years ago

We can change from PEM_read_bio_X509_AUX to PEM_read_bio_X509 where we read the store certificates to use the certificates without trust data.

As previously pointed out this won't work. PEM_read_bio_X509 cannot read the data, which have the BEGIN TRUSTED CERTIFICATE PEM header.

tmds commented 5 years ago

As previously pointed out this won't work. PEM_read_bio_X509 cannot read the data, which have the BEGIN TRUSTED CERTIFICATE PEM header.

Yes, those certificates will be skipped, which is desired.

t8m commented 5 years ago

I do not know what is the situation on SLES or other distributions. But on Fedora/RHEL the data files with and without the trust info contain practically the same certificates except the file with trust info might also contain intermediate certificates which are explicitly distrusted. I do not know off the top of my head if we currently ship such distrusted certificates though, I would have to look.

karelz commented 5 years ago

Fixed in PR dotnet/corefx#32283 for 3.0 and in PR dotnet/corefx#32455 for 2.1 servicing (will be part of 2.1.6 release).

tmds commented 5 years ago

@karelz @bartonjs I'm surprised this is backported. I thought this was only an issue with openssl 1.1 and that .net core 2.1 doesn't support openssl 1.1?

bartonjs commented 5 years ago

It's not OpenSSL 1.1, that was just a timing coincidence

tmds commented 5 years ago

Ah. On Fedora 28, this worked fine for me with .NET Core 2.1 (OpenSSL 1.0) and I only had trouble when running .NET Core 3.0 (OpenSSL 1.1).

osagga commented 5 years ago

if anyone facing this problem on Linux (with openssl 1.1), this fix (https://github.com/NuGet/Home/issues/8169#issuecomment-497591809) worked for me

CosX commented 4 years ago

I experienced this issue when running the restore command as sudo. When I run this command normally, without sudo, I managed to restore as expected. I'm using the 3.0.100 version of dotnet core on Arch.