search

rust-lang / cargo

The Rust package manager
https://doc.rust-lang.org/cargo
Apache License 2.0
12.79k stars 2.42k forks source link

Peer certificate cannot be authenticated with given CA certificates (SSL certificate problem: certificate has expired) #11670

Closed igortavtib closed 1 year ago

igortavtib commented 1 year ago

Problem

I was trying to run my bevy learning project when I get this error after runing cargo run:

error: failed to download from `https://crates.io/api/v1/crates/bevy_derive/0.9.1/download`

Caused by:
  [60] Peer certificate cannot be authenticated with given CA certificates (SSL certificate problem: certificate has expired)

Steps

  1. Init a new project with cargo new
  2. Add dependencies
  3. Run cargo run

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.67.0 (8ecd4f20a 2023-01-10)
release: 1.67.0
commit-hash: 8ecd4f20a9efb626975ac18a016d480dc7183d9b
commit-date: 2023-01-10
host: x86_64-apple-darwin
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.54.0 (sys:0.4.59+curl-7.86.0 system ssl:LibreSSL/2.0.20)
os: Mac OS 10.13.6 [64-bit]
ehuss commented 1 year ago

I believe this is likely due to the age of the system you are running on (Mac OS 10.13.6). Cargo uses the root certificates stored on your system to verify that it is speaking with the correct host. You can check the root certificates by launching the Keychain Access app. crates.io uses the "ISRG Root X1" and "Amazon Root CA 1" certificates. You can check which one is being used by accessing https://static.crates.io/ from your web browser and inspecting the security info. You may also want to double-check that your system clock is correct.

igortavtib commented 1 year ago

Both my certificates (Amazon Root CA 1 and ISRG Root X1) are valid. I can't access https://static.crates.io/ cause I got access denied, I think it Is related to this issue.

ehuss commented 1 year ago

The "AccessDenied" message when visiting https://static.crates.io/ via a browser is normal (the root doesn't have anything to serve).

Perhaps you can post the output of curl --insecure -Iv https://static.crates.io/ That should show what you are connecting to. (Unfortunately it may not show the root certificate.)

igortavtib commented 1 year ago

Thanks for the info about certificates. I finally got it working by running cargo run a few times. I still don't understand the root cause of the error, but if it occurs again, I know where to start investigating. Apologies for misunderstanding the instructions at https://static.crates.io/

weihanglo commented 1 year ago

Go ahead to close this issue in order to make the issue backlog less scary. Thank you for the help of investigation! If you still encounter the issue or have finally figured out the cause, let us know and we can consider re-open.

timmyjose commented 1 year ago

The "AccessDenied" message when visiting https://static.crates.io/ via a browser is normal (the root doesn't have anything to serve).

Perhaps you can post the output of curl --insecure -Iv https://static.crates.io/ That should show what you are connecting to. (Unfortunately it may not show the root certificate.)

Sorry to comment on a closed issue, but I've noticed the same issue for the last few weeks - on some crates (not all), and as @igortavtib mentioned, if you do cargo run enough number of times (regardless of whether cargo update was done or not, apparently), it does go ahead and download. This last bit is what's mystifying (and worrying) to me.

Sharing the output of the curl command here in case it helps:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.1.91:443...
* Connected to static.crates.io (151.101.1.91) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4063 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=fastly-static.crates.io
*  start date: Jan 10 08:14:15 2023 GMT
*  expire date: Apr 10 08:14:14 2023 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7f87e080e200)
} [5 bytes data]
> HEAD / HTTP/2
> Host: static.crates.io
> user-agent: curl/7.79.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 403 
< x-amz-bucket-region: us-west-1
< x-amz-request-id: 1DE7DAM9HKSNB8C5
< x-amz-id-2: 4/rRHejJMWctFhBOmBjtrqvtlSDzpgpEktONXmkYtJplqjD6LuXoHYNtoRWcUUYJ1ZYufSHLCB4=
< content-type: application/xml
< server: AmazonS3
< accept-ranges: bytes
< x-timer: S1678363103.514610,VS0,VE939
< date: Thu, 09 Mar 2023 11:58:23 GMT
< via: 1.1 varnish
< x-served-by: cache-hyd1100027-HYD
< x-cache: MISS
< x-cache-hits: 0
< 

  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Connection #0 to host static.crates.io left intact
HTTP/2 403 
x-amz-bucket-region: us-west-1
x-amz-request-id: 1DE7DAM9HKSNB8C5
x-amz-id-2: 4/rRHejJMWctFhBOmBjtrqvtlSDzpgpEktONXmkYtJplqjD6LuXoHYNtoRWcUUYJ1ZYufSHLCB4=
content-type: application/xml
server: AmazonS3
accept-ranges: bytes
x-timer: S1678363103.514610,VS0,VE939
date: Thu, 09 Mar 2023 11:58:23 GMT
via: 1.1 varnish
x-served-by: cache-hyd1100027-HYD
x-cache: MISS
x-cache-hits: 0

Platform details:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G8012

Cargo details:

$ cargo --version
cargo 1.67.1 (8ecd4f20a 2023-01-10)

I can provide more details if needed, and if it helps!

ehuss commented 1 year ago

@timmyjose Thanks for posting the details.

One explanation for the randomness is that crates.io balances between two different CDNs (fastly and cloudfront), and it may be swapping between the two.

Can you check that you have a root certificate on your system? Open the app "Keychain Access" and search for "ISRG Root X1". You should have a certificate listed there, and its expiration should be June 4, 2035.

timmyjose commented 1 year ago

@timmyjose Thanks for posting the details.

One explanation for the randomness is that crates.io balances between two different CDNs (fastly and cloudfront), and it may be swapping between the two.

Ah, yes. That makes sense indeed.

Can you check that you have a root certificate on your system? Open the app "Keychain Access" and search for "ISRG Root X1". You should have a certificate listed there, and its expiration should be June 4, 2035.

Checked and confirmed! The certificate does exist and has an expiry date exactly as you mentioned:

Screenshot 2023-03-10 at 9 24 19 AM

Thanks for having a look at it, @ehuss, by the way. If you need more information from my side, please do let me know.

timmyjose commented 1 year ago

Just to add an observation from my side - yesterday I uninstalled Rust and reinstalled it again, and faced pretty much the same issue (the issue had been masked presumably because my existing projects all used crates which had been already downloaded previously) - tried it at periodic intervals till it actually worked (again, presumably because the load balancer kicked in and redirected the calls elsewhere).

A secondary observation - if I manually invoke the link (as mentioned in the error message for a specific crate), the browser can download the crate. So I was wondering how much that figures into the equation? I tried poking around at the certificates the browser uses, but seems to be pretty much the same - Amazon CA ultimately.

Another thing I tried is updating my curl version, but that did not help either (since another thread mentioned that Cargo (correct me if I'm wrong) uses curl or libcurl internally? Though I assume that curl/Cargo ultimately default to the system certificate store?).

Do you folks think that this issue should be re-opened? @ehuss @weihanglo

ehuss commented 1 year ago

Yea, seems reasonable to reopen. I don't really have any ideas yet on what might be wrong.

WRT using a browser, browsers have their own set of certificates. Cargo uses the ones managed by your OS on macOS. When reporting, it is very important to include the output of cargo -Vv.

If you want to try to investigate the certificates against your system, I would recommend checking the certificates for:

jdno commented 1 year ago

@timmyjose This line in the curl output stands out to me:

SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

This almost sounds like an old issue with OpenSSL that was reported for Ubuntu (see StackOverflow and Ubuntu bug). The last message in the bug report is from 2014 and mentions OpenSSL 1.0.1, which was afaik the last version in macOS 10.13 before the switch to LibreSSL.

Can you post the versions of curl and openssl on your machine?

curl --version
openssl version

And the following command can be useful for testing. It connects to Fastly using OpenSSL and prints the certificate chain:

openssl s_client -connect fastly-static.crates.io:443 -servername static.crates.io -showcerts </dev/null
timmyjose commented 1 year ago

Hey @jdno, thanks for the suggestions and the SO link (will check it out). In the meantime,

Can you post the versions of curl and openssl on your machine?

$ cargo -vV
cargo 1.68.0 (115f34552 2023-02-26)
release: 1.68.0
commit-hash: 115f34552518a2f9b96d740192addbac1271e7e6
commit-date: 2023-02-26
host: x86_64-apple-darwin
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.54.0 (sys:0.4.59+curl-7.86.0 system ssl:LibreSSL/2.6.5)
os: Mac OS 10.14.6 [64-bit]

(the version of libcurl bundled with Cargo seems to be rather older than the current version, not sure if this could be an issue, but I can try and explore it a bit).

$ curl -V
curl 7.83.1 (x86_64-apple-darwin18.7.0) libcurl/7.83.1 (SecureTransport) OpenSSL/1.1.1o zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 librtmp/2.3 OpenLDAP/2.6.2
Release-Date: 2022-05-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd
$ openssl version
OpenSSL 3.0.0 7 sep 2021 (Library: OpenSSL 3.0.0 7 sep 2021)

And the following command can be useful for testing. It connects to Fastly using OpenSSL and prints the certificate chain:

openssl s_client -connect fastly-static.crates.io:443 -servername static.crates.io -showcerts </dev/null

If I run this command, I do get a bunch of certificates (the whole certificate chain, presumably). 

CONNECTED(00000006)
---
Certificate chain
 0 s:CN = fastly-static.crates.io
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 11 07:34:05 2023 GMT; NotAfter: Jun  9 07:34:04 2023 GMT

-----BEGIN CERTIFICATE-----
<elided>
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
-----BEGIN CERTIFICATE-----
<elided>
-----END CERTIFICATE-----
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
-----BEGIN CERTIFICATE-----
<elided>
-----END CERTIFICATE-----
---
Server certificate
subject=CN = fastly-static.crates.io
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4594 bytes and written 386 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 4BF9E6FD0C293DF700F0F1E69DE39423269A9FC1B832F5772AAC8890E041FD02
    Session-ID-ctx: 
    Resumption PSK: D42657C39DB698535ABA3F078E2A11DDC84EBA80A0DF0C46546567F780E0A59C
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 3600 (seconds)
    TLS session ticket:
    0000 - 60 52 26 63 31 9a bf 58-4c df 94 b4 88 8e c2 8f   `R&c1..XL.......
    0010 - ce 45 fd 61 8c be 15 e6-5a 2c 39 df e0 19 e4 43   .E.a....Z,9....C
    0020 - b7 8f 49 00 fa 6c 85 ba-1e 7b 8e 33 e7 7c 56 6a   ..I..l...{.3.|Vj
    0030 - 3e c5 d6 d4 10 0b 35 7f-30 96 73 f0 2a b9 8a e3   >.....5.0.s.*...
    0040 - 2f 32 12 97 c0 62 50 66-b0 79 62 85 ab eb 28 a1   /2...bPf.yb...(.
    0050 - 79 9d e2 44 c0 dc eb 1d-a5 c1 8c 27 fd f4 c1 8d   y..D.......'....
    0060 - 6b 34 e8 72 87 a3 31 60-f6 44 82 99 5e 05 e4 98   k4.r..1`.D..^...
    0070 - 09 e6 c9 b0 c8 88 17 90-4a 45 76 3b 2b 0f 68 9a   ........JEv;+.h.
    0080 - 99 3d 9f 0b bb 9a 14 d8-0d fc 4c 3d 2f 6c a5 30   .=........L=/l.0

    Start Time: 1678713766
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 8192
---
read R BLOCK

(I've removed the actual text of the certificates in the output above).

ehuss commented 1 year ago

I have figured out how to reproduce this locally.

I'm pretty certain this is because the version of LibreSSL on older versions of macOS is running into the issue where it prefers untrusted certificate chains. From what I understand, the Let's Encrypt certificate is cross-signed with their old (expired) DST Root CA X3 certificate to deal with some bug in older versions of Android.

notes: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

@jdno Is there any way to not have that cross-signature? AFAIK, it is not feasible to update the system libcurl or libressl. Are there any known workarounds if you are on an older version of OpenSSL?

jdno commented 1 year ago

Yeah, looking at the version numbers only system ssl:LibreSSL/2.6.5 seems really old.

I don't think we're using the certificate chain that is cross-signed with DST Root CA X3. The serial number of the returned ISRG Root X1 certificate matches the serial number of the self-signed certificate and the DST Root CA X3 root is not included in the certificate chain.

🤦‍♂️ My bad, totally misinterpreted the console output and the certificate details in the browser.

Let me check with our contact at Fastly what we can do.

timmyjose commented 1 year ago

Hey @ehuss @jdno, thanks for looking into the issue. Appreciate it. If there's anything I can do from my side, please do let me know. I'm not familiar with Cargo internals, but in case you need any data, or need me to try out updating/installing some components, I could try it out. (I tried upgrading LibreSSL and making it the default openssl client, but that didn't help - how does Cargo pick up the versions specified in the Cargo -vV output?).

jrndinn commented 1 year ago

Same exact issue on my side using macOS 10.14.6. ISRG Root X1 also expires June 4th 2035 and Amazon Root CA 1 January 17th 2038. Cargo switches between working and not working periodically. If there is anything I can do to gather more data please let me know

curl --version

curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

openssl version

LibreSSL 2.6.5

cargo -vV

cargo 1.70.0-nightly (7d3033d2e 2023-03-08)
release: 1.70.0-nightly
commit-hash: 7d3033d2e59383fd76193daf9423c3d141972a7d
commit-date: 2023-03-08
host: x86_64-apple-darwin
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.54.0 (sys:0.4.59+curl-7.86.0 system ssl:LibreSSL/2.6.5)
os: Mac OS 10.14.6 [64-bit]
dominiqs81 commented 1 year ago

Same issue and very same setup as @jorundev (Mac OS 10.14.6 with LibreSSL 2.6.5). The error systematically happens when trying to upgrade rust from 1.66 to 1.68 via homebrew.

jdno commented 1 year ago

Apologies for the inconvenience, folks. Please know that I'm working on this.

I did some more research and testing yesterday, and eventually found this post by a member of Let's Encrypt's staff that confirms @ehuss suspicion that the issue has to do with the cross-signed root certificate. Quickly summarized, OpenSSL < 1.1 and LibreSSL < 3.2 will just not work with Let's Encrypt's default chain.

I'm now in the process of switching our staging environment over to a certificate provided by GlobalSign. If we can confirm that this resolves the issue with older OpenSSL versions, we'll roll it out to production as well.

jdno commented 1 year ago

Sorry for the wait, everyone! It took a few days to confirm the fix on staging and test the rollout process to production.

Our Fastly service for static.crates.io now serves a certificate signed by Global Sign.

This has resolved the above issues on macOS 10.13 (or other systems that still rely on OpenSSL < 1.1) in my test environment. If you continue to experience issues, please let me know so that we can investigate.

timmyjose commented 1 year ago

Sorry for the wait, everyone! It took a few days to confirm the fix on staging and test the rollout process to production.

Our Fastly service for static.crates.io now serves a certificate signed by Global Sign.

This has resolved the above issues on macOS 10.13 (or other systems that still rely on OpenSSL < 1.1) in my test environment. If you continue to experience issues, please let me know so that we can investigate.

Thank you so much for the fix, @jdno. Also thanks to @ehuss! Really appreciate it.

I just tested it now (had to wait a bit till the GlobalSign one got picked):

~/dev/playground:$ curl --insecure -Iv https://static.crates.io/ 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.1.91:443...
* Connected to static.crates.io (151.101.1.91) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2853 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=static.crates.io
*  start date: Mar 27 10:19:44 2023 GMT
*  expire date: Apr 27 10:19:43 2024 GMT
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2023 Q1
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: static.crates.io]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fa5eb801400)
} [5 bytes data]
> HEAD / HTTP/2
> Host: static.crates.io
> user-agent: curl/7.83.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [185 bytes data]
< HTTP/2 403 
< x-amz-bucket-region: us-west-1
< x-amz-request-id: 36MBENX6XFKQ1HK2
< x-amz-id-2: d+ittT5zQRviYW9eaJGbrMaBey2611pMNavnmV9YPCBGWza0aiXARhGvNxH0RfRaEqPW8OGDjDhTxOBdW02XZg==
< content-type: application/xml
< server: AmazonS3
< accept-ranges: bytes
< x-timer: S1679979033.363817,VS0,VE928
< date: Tue, 28 Mar 2023 04:50:34 GMT
< via: 1.1 varnish
< x-served-by: cache-hyd1100022-HYD
< x-cache: MISS
< x-cache-hits: 0
< 

  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Connection #0 to host static.crates.io left intact
HTTP/2 403 
x-amz-bucket-region: us-west-1
x-amz-request-id: 36MBENX6XFKQ1HK2
x-amz-id-2: d+ittT5zQRviYW9eaJGbrMaBey2611pMNavnmV9YPCBGWza0aiXARhGvNxH0RfRaEqPW8OGDjDhTxOBdW02XZg==
content-type: application/xml
server: AmazonS3
accept-ranges: bytes
x-timer: S1679979033.363817,VS0,VE928
date: Tue, 28 Mar 2023 04:50:34 GMT
via: 1.1 varnish
x-served-by: cache-hyd1100022-HYD
x-cache: MISS
x-cache-hits: 0

and I was able to fetch and build with no problems:

~/dev/playground/foo:$ cargo build
  Downloaded phoron_asm v1.0.2
  Downloaded 1 crate (94.7 KB) in 2.03s
   Compiling phoron_core v0.5.4
   Compiling phoron_asm v1.0.2
   Compiling foo v0.1.0 (/Users/z0ltan/dev/playground/foo)
    Finished dev [unoptimized + debuginfo] target(s) in 8.93s

Thank you!

jdno commented 1 year ago

After running the new certificates in production for four weeks, I'm gonna close this issue. If anyone runs into this or a similar issue in the future, feel free to create a new issue.