search

oerdnj / deb.sury.org

Public bugreports for anything ppa:ondrej/*
825 stars 26 forks source link

Wrong Let's Encrypt CA chain presented by your server #1792

Closed lastuptodate closed 2 years ago

lastuptodate commented 2 years ago

Frequently asked questions

Describe the bug packages.sury.org web server is presenting wrong CA chain see: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

To Reproduce Steps to reproduce the behavior:

openssl s_client -connect packages.sury.org:443
Certificate chain
 0 s:CN = packages.sury.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

LE

# update-ca-certificates 
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
# apt update
Hit:1 http://repo.clacos.ninja buster InRelease
Ign:2 https://packages.sury.org/php buster InRelease                                                                                                                                                              
Err:3 https://packages.sury.org/php buster Release                                                                                                                                                                
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification.

Expected behavior The certicate must be recreated with the right CA chain, see: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ This is generaly the case when you are using an outdated certbot provided by Debian rather than certbot by https://certbot.eff.org/. In some cases, you have to delete+create cert rather than renew. Also, --prefered-chain may also help you.

Distribution (please complete the following information):

oerdnj commented 2 years ago

I don't really see the behaviour described here:

ondrej@kage:~$ gnutls-cli packages.sury.org:443
Processed 189 CA certificate(s).
Resolving 'packages.sury.org:443'...
Connecting to '185.152.64.17:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=packages.sury.org', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0x0391013a17469f7f084671ef20c765694d6e, RSA key 2048 bits, signed using RSA-SHA256, activated `2022-05-22 10:10:15 UTC', expires `2022-08-20 10:10:14 UTC', pin-sha256="jgwPBRVN5iMySczBW4m1VqMzCmw/ETYRRT2jL+pBZXA="
    Public Key ID:
        sha1:ed56dceb8acbaa62da327a6400abcc40b66540e7
        sha256:8e0c0f05154de6233249ccc15b89b556a3330a6c3f113611453da32fea416570
    Public Key PIN:
        pin-sha256:jgwPBRVN5iMySczBW4m1VqMzCmw/ETYRRT2jL+pBZXA=

- Certificate[1] info:
 - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
 - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is trusted.
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID: 6F:CD:59:92:B7:AA:8C:1B:3D:07:E4:F2:A0:61:65:9E:F8:DB:5D:3F:F7:64:B9:75:9C:AC:A5:E5:B4:8E:C7:2A
- Options:
- Handshake was completed

- Simple Client Mode:

- Peer has closed the GnuTLS connection
$ openssl s_client -connect packages.sury.org:443
CONNECTED(00000005)
depth=3 C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = *.b-cdn.net
verify return:1
---
Certificate chain
 0 s:/CN=*.b-cdn.net
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
---
Server certificate

So, what is the DNS records and IP address(es) you are connecting to?

oerdnj commented 2 years ago

Nor with SNI:

$ openssl s_client -connect packages.sury.org:443 -servername packages.sury.org
CONNECTED(00000005)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = packages.sury.org
verify return:1
---
Certificate chain
 0 s:/CN=packages.sury.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISA5EBOhdGn38IRnHvIMdlaU1uMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA1MjIxMDEwMTVaFw0yMjA4MjAxMDEwMTRaMBwxGjAYBgNVBAMT
EXBhY2thZ2VzLnN1cnkub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAlLjyBAnQJuBcJpJqgOMmAkNSK8yigld1prP4wljNoGnYXuRWQ3uPcpcSCeQC
yT3b5xg/ZkQG0KtSlIZCV6R/8zqGFU/tj0wqZYh6sOxKzJGjIPPjbHRu9aZ/LgT8
f1GECT42+SDarTkDlX16Oggj3q69hEFgsAs9QF4eb8mSINi6X8oKW8vcYVElWZyd
c8WbkvSUwtwEqKLpXPkNvECloReANeyVCNZ0eh48PQOvwQ8vkwglojyerNpK85Rm
mHbDj6/4TMp9NwwfW50BYqL/787c2iHShyxcqAl8pPVXnTmuznNpMEm5OVa8Ch4M
v+FSfonzYy4+d8vZvtcDcF2VBwIDAQABo4ICTTCCAkkwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBQy43kyKEjVoswPhP2fTqIYhOCsbzAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghFwYWNrYWdlcy5zdXJ5Lm9yZzBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB3AEHIyrHf
IkZKEMahOglCh15OMYsbA+vrS8do8JBilgb2AAABgOt17W4AAAQDAEgwRgIhANfo
CD2reLy+rGOLbEzJRKkTYm/PMO/bJkSjC5gw4PzyAiEAgu8xyAS79AW0AZNjAP+b
fepkRRKpLzWTL2kHVNXx7N4AdgApeb7wnjk5IfBWc59jpXflvld9nGAK+PlNXSZc
JV3HhAAAAYDrde1qAAAEAwBHMEUCIETybellG1YyDh6YXLohH4gzoAcLVuMCtR/p
aYRuNZ6IAiEA0iQ6N7NyLQAm/MoedLiqXTUIPEmcgyp5hGA/3QW1HZswDQYJKoZI
hvcNAQELBQADggEBAKW/uGoAHFUGVsqOb1mU6Jnc9psr5d4HlJVvXPSut/F3gyM3
+5JW0UJyG3KyPatlHVnEvdbBu3YF8aUxUsoFB+IKtBEL+GIKxyvxU1HojDYM2Lt6
TZt6RRlPVoWxIgSOX8jcQdvreJawTI7i8AOXJ59ZgK5DsLmzA0dXJfQYdFQAeEQX
0dtE7Qf4y1I5ZKHQhrydoTpObuk3qkT2GSN1CrQC9jHmHGjEX5ReN1V7Mo9c5hp/
opieB4jTpVhnNNQBKtNGIke8ySdGRoOIrOiCITrOMkE/sxIwzx0PE2Afjp2OWyh5
shQBHJJslbSODPNyOQsQDxnLdmnsumnU0XoTzJI=
-----END CERTIFICATE-----
subject=/CN=packages.sury.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 4666 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 39FD7FAC3DB1FBD7265266E55B982E5F057AAB1E02D66A952276AE77D2D39507
    Session-ID-ctx:
    Master-Key: D01418236C4F792C43DE402B14DC8EE0794816990BF0FECACEEDC3FBD83D92724F40AEF848A8E919517C000B9A4915E1
    TLS session ticket lifetime hint: 43200 (seconds)
    TLS session ticket:
    0000 - de af 11 c7 7f df 57 0b-7e d8 66 63 ab ac 3e e7   ......W.~.fc..>.
    0010 - 8c e6 17 5a 5c f2 59 2d-1c f9 93 11 0e 12 2e 7d   ...Z\.Y-.......}
    0020 - 26 f8 6a ef 02 4a 33 a6-d3 d9 59 71 a1 f7 4b 59   &.j..J3...Yq..KY
    0030 - 42 a4 91 40 2b 98 08 23-9c cb 28 bb 8b 86 6d b8   B..@+..#..(...m.
    0040 - d6 fd 7e 94 61 e6 a7 9f-1d e6 14 4b 0d 01 7f 32   ..~.a......K...2
    0050 - 71 6f 30 62 b0 bc 55 e8-1a 77 5f 0b 77 d7 58 86   qo0b..U..w_.w.X.
    0060 - 35 b4 d4 cc 53 4b 4f 9f-91 4b 7c fb 6a 03 15 ff   5...SKO..K|.j...
    0070 - ef 98 1d 91 43 e4 80 98-31 f4 8f 7c d2 e5 83 b0   ....C...1..|....
    0080 - a2 53 25 17 ac 85 51 1b-28 31 e6 a9 a0 45 f4 43   .S%...Q.(1...E.C
    0090 - 34 37 8b 66 4e f4 1a 00-8d 53 3b 89 c1 d6 64 68   47.fN....S;...dh
    00a0 - 53 b1 b6 89 52 74 75 dd-e5 6b f3 89 bf 7d 60 95   S...Rtu..k...}`.
    00b0 - eb d6 ed 52 6e 11 db 67-28 59 b9 8a 6f 79 60 d8   ...Rn..g(Y..oy`.

    Start Time: 1655364564
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
oerdnj commented 2 years ago

Neither there's anything wrong at the SSLLabs: https://www.ssllabs.com/ssltest/analyze.html?d=packages.sury.org

There's trusted Certification Paths listed for every client.

oerdnj commented 2 years ago

Nevertheless, I opened the ticket with @BunnyWay to remove the cruft being sent as part of the certificate chain.

However, if the validation fails on your side, it means you are missing:

ISRG Root X1 Self-signed Fingerprint SHA256: 96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6 Pin SHA256: C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M= RSA 4096 bits (e 65537) / SHA256withRSA

in the trust store.

oerdnj commented 2 years ago

But perhaps it's intentional - if you read the link that you sent yourself:

DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates. There’s one important exception: older Android devices that don’t trust ISRG Root X1 will continue to work with Let’s Encrypt, thanks to a special cross-sign from DST Root CA X3 that extends past that root’s expiration. This exception only works for Android.

I don't really care about the Android devices, but other CDN users might.

lastuptodate commented 2 years ago

Let's encyrpt has changed CA chain

In this example, we are using LE with a unique and good CA chain path. This is the recommanded solution

sleemanj commented 2 years ago

@oerdnj is correct, it's intentional, this is the cross-signed chain that servers should serve up so that (older) Android doesn't get lost (with that said, how many Androiders are going to be pulling the repo).

Submitter must be missing the ISRG root, which they can get here... https://letsencrypt.org/certs/isrgrootx1.pem

todeveni commented 2 years ago

@lastuptodate What's your ca-certificates package version?

buster without ca-certificates installed: - Status: The certificate is NOT trusted. The certificate issuer is unknown.

buster with latest ca-certificates 20200601~deb10u2 installed: - Status: The certificate is trusted.

oerdnj commented 2 years ago

@lastuptodate Well, you are wrong. The very material you provided disputes what you are saying. There's nothing wrong with the certificate chain sent from the server. Thanks @sleemanj for confirming my hypothesis.

lastuptodate commented 2 years ago

Have a look at CA Chain, they are both using Let's Encrypt: https://www.hardenize.com/report/packages.sury.org/1655367513#www_certs https://www.hardenize.com/report/vdp.gong.io/1655367866#www_certs

oerdnj commented 2 years ago

Have a look at CA Chain, they are both using Let's Encrypt: https://www.hardenize.com/report/packages.sury.org/1655367513#www_certs https://www.hardenize.com/report/vdp.gong.io/1655367866#www_certs

https://github.com/oerdnj/deb.sury.org/issues/1792#issuecomment-1157363230

lastuptodate commented 2 years ago

Thank for your help and your time, I think sadly that i will not convince you so I will resolved this problem using this sad and worst solution:

Acquire::https::packages.sury.org::Verify-Peer "false";
Acquire::https::packages.sury.org::Verify-Host "false";
oerdnj commented 2 years ago

The hardenize.com report is plain wrong, it shows just the second chain. I've sent you the ssllabs.com report that you chose to ignore that shows both certificate paths.

It's your choice to ignore the facts and advice what's wrong with your system, so don't put it on me.

sleemanj commented 2 years ago

@lastuptodate

If you remove the expired X3 root from your system, and have the valid ISRG root on your system, then I expect it might verify.

You just need one of them to verify, if as you say your tools are picking the X3, then you want to convince them to pick the ISRG, since the X3 is no use to you on your system (but is to old Androiders on their system).

I think, from vague recollection, that's what I did back last year, on my ancient 14.04 (not for pulling this repo obviously, but to shut up some other stuff).

Indeed, looking in /etc/ca-certificates.conf on that 14.04 system I have the X3 cert disabled.

todeveni commented 2 years ago

@lastuptodate

I'd still like to add, that your local certificate store is old and/or badly maintained.

There's zero problems with clean Buster docker image using the instructions in README.txt, which installs latest ca-certificates package.

lastuptodate commented 2 years ago

I can't also reproduce on a fresh Buster but I can on old Buster install. So I try to see in "apt list --upgradable" packages linked to apt/curl or ssl.

And the winner is:

Unpacking libgnutls-dane0:amd64 (3.6.7-4+deb10u7) over (3.6.7-4+deb10u3) ...
Unpacking libgnutls30:amd64 (3.6.7-4+deb10u7) over (3.6.7-4+deb10u3) ...

Upgrading this 2 packages solved my problem. Thank you all and really really really sorry and all my apologies for the noises about that

akaPipo commented 2 years ago

I have fixed it this way https://github.com/oerdnj/deb.sury.org/issues/1729#issuecomment-1157442150