Cloudflare changes with Let's Encrypt

[aterrel]

Email to Bryan V (Bokeh) from Cloudflare

We are reaching out to inform you about an upcoming change that will impact the device compatibility of Let’s Encrypt certificates issued after May 15th, 2024. We are reaching out to you because we identified that you are currently using Let’s Encrypt certificates through Universal SSL, Advanced Certificate Manager, Custom Certificates, or SSL for SaaS. We recommend that you familiarize yourself with the Let’s Encrypt change and make any necessary adjustments ahead of time. I don't really know what to do with this information (or if it actually affects the bokeh.org domain, or if it just emailed me because I am on the NF CF account in general) is there someone I can forward this to? (edited)

Need to investigate and see if there is any impact to our users.

[bryevdv]

FYI here is the full email

> Hi, > > We are reaching out to inform you about an upcoming change that will impact the device compatibility of Let’s Encrypt certificates issued after May 15th, 2024. We are reaching out to you because we identified that you are currently using Let’s Encrypt certificates through Universal SSL, Advanced Certificate Manager, Custom Certificates, or SSL for SaaS. We recommend that you familiarize yourself with the Let’s Encrypt change and make any necessary adjustments ahead of time. > > **Change Overview** > Let’s Encrypt issues certificates through two chains: the ISRG Root X1 chain and the ISRG Root X1 chain cross-signed by IdenTrust’s DST Root CA X3. The cross-signed chain has allowed Let’s Encrypt certificates to become widely trusted, while the pure chain developed compatibility with various devices over the last 3 years, growing the number of Android devices trusting ISRG Root X1 from 66% to 93.9%. > > Let’s Encrypt [announced](https://links1.cloudflare.com/u/click?_t=4d37ee5dc20f46da9035f25c66794274&_m=9002f57b8d81426b8246d2ba1b074007&_e=o9IhLLXXRMDOsoADvPd-gmjMbWx4EQtVQHIf34W4tQfns26DVp0l1h4jao4mD9vzWV8ZinzTT8Ofhdk0LYdQhXAoiln2vWazKIBylrJv3tKo5lFDbuQWNPJf-PJT_VC2p3iPr2I76uSUzig43etzch0hvWor34ej82CqGhseDUcyX3T1suIwpo8KmWAFeYPVnqQNNSo1ox5HZ_IoUXV_YaTX4LUjdNagaOM36mPEB1ostOhTgKpHe_uuXu0pgQbigM_URv0WVmPcmcmAL1QD8yuN8P3L7dUgcpfjXtUiEkwarjNiEfuYHuavT_hhiia5heWVuMvhCLloCfWjnM8yYaVbEBbfklKOBqTgR0IC7uw%3D) that the cross-signed chain is set to expire on September 30th, 2024. As a result, Cloudflare will stop issuing certificates from the cross-signed CA chain on May 15th, 2024. > > **Impact** > The expiration of the cross-signed chain will primarily affect older devices (e.g. Android 7.0 and earlier) and systems that solely rely on the cross-signed chain and lack the ISRG Root X1 chain in their trust store. This change could result in certificate validation failures on these devices, potentially leading to warning messages or access problems for users visiting your website. > > **Impact to certificates issued through Universal SSL, Advanced Certificate Manager, or SSL for SaaS:** > To prepare for the CA expiration, after May 15th, Cloudflare will no longer issue certificates from the cross-signed chain. Certificates issued before May 15th will continue to be served to clients with the cross-signed chain. Certificates issued on May 15th or after will use the ISRG Root X1 chain. Additionally, this change only impacts RSA certificates. It does not impact ECDSA certificates issued through Let’s Encrypt. ECDSA certificates will maintain the same level of compatibility that they have today. > > **Impact to certificates uploaded through Custom Certificates:** > Certificates uploaded to Cloudflare are bundled with the certificate chain that Cloudflare finds to be the most compatible and efficient. After May 15th, 2024, all Let’s Encrypt certificates uploaded to Cloudflare will be bundled with the ISRG Root X1 chain, instead of the cross-signed chain. Certificates uploaded before May 15th will continue to use the cross-signed chain until that certificate is renewed. > > **Important Dates** > > **May 15th, 2024**: Cloudflare will stop issuing certificates from the cross-signed CA chain. In addition, Let’s Encrypt Custom Certificates uploaded after this date will be bundled with the ISRG X1 chain instead of the cross-signed chain. > > **September 30th, 2024**: The cross-signed CA chain will expire. > > **Recommendations:** > To reduce the impact of this change, we recommend taking the following steps: > **Change CAs**: If your customers are making requests to your application from legacy devices and you expect that this change will impact them, then we recommend [using a different certificate authority](https://links1.cloudflare.com/u/click?_t=4d37ee5dc20f46da9035f25c66794274&_m=9002f57b8d81426b8246d2ba1b074007&_e=o9IhLLXXRMDOsoADvPd-gmo8iHEBi97rwsjmOOFUvQ03X1W8vZJOU0ZrY8gdQZB_eQyWPN61QmwV1LxKG_d-RMBTNrp_fcB811caRyIVlYFOGt4udE-Fw55aW4SUMwQ7AYpFMKKAy2SX3PGCHxkD5y14t0u97KyC6K6H4zPdTB1aaZvNIckFqmj-6kFAYigpEFwNBaT2rmtLOy_UokpV8VYfSnsh3-iCgfVNuESNbeu1HI5-S7o_pbpTGhlPR1c3b2-ONHUGnLd1na8hJ_6_xbEhMJoCW3PGKDst_ld1aOFRcJ4pVdn8F5gusldyo_SitGJ2i-QPu_DRGYtFcc_x_dbMpjHYI94VjsUOJ7OraisxJG_uquTzw64I4-LA4kV8) or [uploading a certificate](https://links1.cloudflare.com/u/click?_t=4d37ee5dc20f46da9035f25c66794274&_m=9002f57b8d81426b8246d2ba1b074007&_e=o9IhLLXXRMDOsoADvPd-gmo8iHEBi97rwsjmOOFUvQ03X1W8vZJOU0ZrY8gdQZB_eQyWPN61QmwV1LxKG_d-RBt-uSaQSnmstbTMRW2wMZqWCDNgpeNMBkjOTzrJuTbuhGSpUHDUX_2bI9HLLZ0wAsDKt2HmfNjS0qHSFZ0e84J0sHOEJ1m4NgIQ2tPiYLJpkOjqm14N_qWaYSZ9dbglOmz35vrE42qQU8vtu1WwmjvXi-_wZ6A9W5Ir0QQcI1Q9YouGrdrx-OPtz2FEL-PnsPn2AaJJplC3ijQJ_ODVymlcvLuZtYyO8o7inHJxR4bxJsEPR4iHx3-olM1wRg-nCvgPbtaTDDQ3EhXpnPuSqXYPFjmDF8DgZts_n-Cupjyf) from the CA of your choice. > **Monitoring**: Once the change is rolled out, we recommend monitoring your support channels for any inquiries related to certificate warnings or access problems. > **Update Trust Store:** If you control the clients that are connecting to your application, we recommend upgrading the trust store to include the ISRG Root X1 chain to prevent impact. > If you have any questions, we recommend that you refer to our [Developer Documentation](https://links1.cloudflare.com/u/click?_t=4d37ee5dc20f46da9035f25c66794274&_m=9002f57b8d81426b8246d2ba1b074007&_e=o9IhLLXXRMDOsoADvPd-gmo8iHEBi97rwsjmOOFUvQ0CQu1AHmRnJFLB4_VqysimOMibWudmw0JqPly-NFtFRKBDxpu2_2UiYUpflpedW8Hp3YMICiEYgEe4yyFBXGIvIiHNjYbFZXS-Yn5FqHRuINtrk95Q7526yRZlAhN9GPk8M1r6BZQVkLms3J_wO-9dD7h5ziuRjFzrPCHHRzvTWtopC3wRW9YZSKcEbO0ddgrJEV5iUpD1PIpude7EEAeg328yxhv7OHsIPRvE4rsdmdBtc0WseKYxaWCsARY0daJz4_R2if3WY4ynIVeVwq-i66l6w73rUOhrk-DtF_1gTB6O0R7k_54ezoGZ33H6OncG7q2CwJAN189Up96Ueahb) or [blog post](https://links1.cloudflare.com/u/click?_t=4d37ee5dc20f46da9035f25c66794274&_m=9002f57b8d81426b8246d2ba1b074007&_e=o9IhLLXXRMDOsoADvPd-grSBzMOSBpS_X7zDhfin7bwp2wekBAN4JnAk7K3TrUCAHQexn-k1cjZRUBGbRI46VyTLyOxBH38HE5Npnkby68lfTONlpf_aA34A-5wxplWUYIHFGKfSS4-wLqHNQD5WBYVwJy1rzxkmdV-zW5dNutNi9ScrcmjvBnrn9bLNWGUw3Gk7LTiDb1zYcn-ape_l_fsTiIjG3XraEy6sh6CjavHuDZJbotH41i0uhKxsDk_8yIjHOMGCoh37AKj_34YxZo2Tdi38pIYG1ohVXudthSVju59lvcZu2e-kaocvWUTpFCQrgskhI6EmklrwUJjwJORkBRE4E47Q7pfgUREe5ZtUy_O955L3ZRxEbYE4-CdYOr1ZqFq7_sidHX8Bp9fvOuK2OJWs5k0WrORbKkDufdc%3D) regarding this change. If you are an Enterprise customer and have additional questions or concerns, please reach out to your Account Team.

Headers:

``` Delivered-To: bryan@bokeh.org Received: by 2002:a05:7000:57cf:b0:55d:8256:f125 with SMTP id v15csp205529mau; Fri, 15 Mar 2024 08:00:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGwYi8bdGWMLXrnc6UQ7kg8ja6PbF9LGi0NjT4/yjRdrrYEoBZr033Ap33bp6J20qsM3HvJ X-Received: by 2002:a05:6358:1282:b0:17b:f880:a3c1 with SMTP id e2-20020a056358128200b0017bf880a3c1mr5769497rwi.17.1710514802570; Fri, 15 Mar 2024 08:00:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1710514802; cv=none; d=google.com; s=arc-20160816; b=dT77DDwhChDKYhB8vUP/qlxrmPVgrgmh0VxcfFCPCvL9qhTm0o5NEdc7bm5aJn1t2j xY4AlRjPZ2Qgk2HzRZX5R+9Z2VQi/h7zGz0CC8P457n3x6W9qa5H7Wzp2f+k+r5olI9H KjfZaG5mkR5bQiyesqzQZEL0I8/1+aho87W5xprHqXQp4D1t5yAlh3pkwpBzfAtWFE7X r0UKRhmxz94eMbwATdEhGHxrURn+qye4MU6CnnTSc199nH3meaCAfS2nSYcmkF4cxmfb PWQIkOA1Lf+CFbmuC0U1Wi04B1V/3mZDC0kkQFSgCwaYWrxUmMvJw9WP2N5Mp8RDifRe yZKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:subject:from:reply-to:mime-version:date:message-id:to :dkim-signature; bh=D20TInOxZyOnvUf7ANmmbmwA0uMX6W0XolUkRDcVGiA=; fh=8fd6yLHyybGtzuWgxEKHjZ6WygvTiq0Gv0YUTtexX6c=; b=we9SQuNpttlX/WmVL7wmre/3jF34T9MwKWERYo2iz298wdvqYLGxoNN0FMgEi4Bu/Y +Wh2uJpcazyao6NE5HlPJrlLyBLfnNJan5ocRb4sey7TtF3TTOvy4p6SHkC2M3uMtMoF kn4H0zFpZBYAqTgDN4fzWiveBSrKB87ZuMIwxXnNkLer9jzLtB/zPKr55affrLrwh+xq 5aLZu3PkakWbkT4MMnlxhhF0TVqaovbhEl4T91YxMz39U4nHMQmsIXmqK45VjWmU3iGI qz2wI9pb9k3BgakUG0bvwx00POjdDMXxiQOmXidM8Qtk/2/Sh2dSHksC6zy3urAWA4YJ xpXg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@em1.cloudflare.com header.s=scph0124 header.b=t+9svig2; spf=neutral (google.com: 156.70.53.53 is neither permitted nor denied by best guess record for domain of msprvs1=19804oio6wj8p=bounces-280172-399@bounce1.cloudflare.com) smtp.mailfrom="msprvs1=19804oiO6WJ8P=bounces-280172-399@bounce1.cloudflare.com"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=cloudflare.com Return-Path: Received: from mta-70-53-53.sparkpostmail.com (mta-70-53-53.sparkpostmail.com. [156.70.53.53]) by mx.google.com with ESMTPS id a186-20020a6390c3000000b005d8b313de26si2733605pge.594.2024.03.15.08.00.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Mar 2024 08:00:02 -0700 (PDT) Received-SPF: neutral (google.com: 156.70.53.53 is neither permitted nor denied by best guess record for domain of msprvs1=19804oio6wj8p=bounces-280172-399@bounce1.cloudflare.com) client-ip=156.70.53.53; Authentication-Results: mx.google.com; dkim=pass header.i=@em1.cloudflare.com header.s=scph0124 header.b=t+9svig2; spf=neutral (google.com: 156.70.53.53 is neither permitted nor denied by best guess record for domain of msprvs1=19804oio6wj8p=bounces-280172-399@bounce1.cloudflare.com) smtp.mailfrom="msprvs1=19804oiO6WJ8P=bounces-280172-399@bounce1.cloudflare.com"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=cloudflare.com X-MSFBL: wDo7dOB6U7XMLaqKxBS2SH5VDefmiqFzIuDxpYcXtEI=|eyJyIjoiYnJ5YW5AYm9 rZWgub3JnIiwibWVzc2FnZV9pZCI6IjY1ZWY3MTYyZjQ2NTQ0YWFkOTAyIiwiY3V zdG9tZXJfaWQiOiIyODAxNzIiLCJzdWJhY2NvdW50X2lkIjoiMzk5IiwidGVuYW5 0X2lkIjoic3BjIn0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=em1.cloudflare.com; s=scph0124; t=1710514801; i=@em1.cloudflare.com; bh=D20TInOxZyOnvUf7ANmmbmwA0uMX6W0XolUkRDcVGiA=; h=To:Message-ID:Date:Content-Type:From:Subject:From:To:Cc:Subject; b=t+9svig2ebYb3MZdZUidOzf/nNwiTk/cCIKaybVIXT+6tU5cy25Q1sXW/Ha3rpCoz S5Ve7MlTGiGP7sKqnEhgM3TxCrvI1d0qtpDoFSzW2QVbQvoBv91PJOmiZmCK8un7Z/ pp0O2WfXkWwL6zPaJ+xpNreAaqGgqco0adTnyEyo= To: bryan@bokeh.org Message-ID: <9D.20.34918.17264F56@iw.mta1vrest.cc.prd.sparkpost> Date: Fri, 15 Mar 2024 15:00:01 +0000 Content-Type: multipart/alternative; boundary="_----YmJgEM6KdKj2lYRb0nwang===_D8/10-34918-17264F56" MIME-Version: 1.0 Reply-To: em@em1.cloudflare.com X-Campaign-ID: 9296192 From: "Cloudflare" Subject: [Cloudflare - Action Required] Upcoming Let's Encrypt certificate chain change X-Message-ID: 9002f57b8d81426b8246d2ba1b074007 X-Feedback-ID: 12292333:9296192:46499:iterable Feedback-ID: 12292333:9296192:46499:iterable ```

[aterrel]

Seems like this will affect a minority of users so we will just monitor and change CAs if required after the switch in May.

[aterrel]

Have not heard of any consequences. Will leave up for another month.

[ivirshup]

I think this might be affecting scverse.org? Our site is down due to ssl certificate and we were recently moved to cloudflare

[aterrel]

@ivirshup It looks like scverse.org was set up to do encryption all the way to your server. I switch to only encryption to cloudflare and unencrpyted to your server and the page is now reachable. If you want encryption all the way to your server we will need to get your keys figured out so it's the correct key going through cloudflare.

[ivirshup]

Thanks so much! I'm still a little confused about what happened to make this stop working/ why it was working before. Was cloudflare always encrypting the whole thing, and did that work before?

Right now on our GitHub pages settings page I see that there's an error getting a TLS certificate (which stays even after I try restarting the process):

Which looks like it could be addressed by changing the records to not be proxied? Though I unfortunately don't think I have access to the records since we moved registrar from Namecheap to cloudflare.

Partially related, it looks like another of our subdomains (muon.scverse.org) is getting "too many redirect" errors. This had come up when we first moved to cloudflare but @martey fixed by "telling [cloudflare] not to send insecure HTTP requests".

I'm starting to suspect this is unrelated to the cloudflare change at the top of this issue, and was just due to our GitHub pages letsencrypt certificates expiring once we had switched to cloudflare.

[aterrel]

well the redirects happen when you redirect http requests to https. I'll have to find some time to debug, but if you can turn off https redirect on muon.scverse.org it should work :)

[flying-sheep]

Is HSTS configured for the two sites? Then the HTTP→HTTPS redirect is no longer that necessary.

But without HTTPS, people typing scverse.org into their browser will send a HTTP request that needs to be redirected.

[aterrel]

Moving this ticket to #40