[INFRA-3083] Certificate for pkg.jenkins.io appears to have expired

[jenkins-infra-bot]

The cert seems to have expired for pkg.jenkins.io.

Not sure if this is related to the Let's Encrypt cert expiration that was scheduled for today.

This is on a fresh Ubuntu 20.04.2 LTS ec2:

Err:5 https://pkg.jenkins.io/debian-stable binary/ Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 151.101.54.133 443]
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
E: The repository 'http://pkg.jenkins.io/debian-stable binary/ Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

---

Originally reported by gcarnell, imported from: Certificate for pkg.jenkins.io appears to have expired

• assignee: dduportal
• status: Closed
• priority: Blocker
• resolution: Won't Fix
• resolved: 2021-10-15T09:57:11+02:00
• imported: 2022/01/10

[jenkins-infra-bot]

JIRAUSER134120:

I'm not sure where the issue is entirely, but I ran into this too, and what worked for me was to edit /etc/ca-certificates.conf and add a !to the line that is `mozilla/DST_Root_CA_X3.crt` so that it looks like `!mozilla/DST_Root_CA_X3.crt` - then run `update-ca-certificates`

[jenkins-infra-bot]

JIRAUSER130237:

Steev Klimaszewski
Thanks, that seems to work for me too.

[jenkins-infra-bot]

JIRAUSER134144:

I am also facing same issue.

But this solution worked for me.

[jenkins-infra-bot]

JIRAUSER134116:

Yes, I was able to use the workaround described above to get past this, but I think I'll leave the ticket open as the underlying problem still exists.

[jenkins-infra-bot]

nfalco:

just `sudo yum install ca-certificates` solve my issue.

[jenkins-infra-bot]

darkwizard242:

Second that, upgrading the ca-certificates package allows the handshake to be successful. Appears that repo likely is using a newer version of the ca certificate and most other systems just need to have ca-certificates package upgrade for fetching the latest version of certificates.

[jenkins-infra-bot]

JIRAUSER133977:

I am also facing same issue, but install the ca-certificates cannot solve my issue, my os using RHEL 8 

when I using openssl connect to pkg.jenkins.io and the return is fine (1), but when using nslookup the resolved address located to fastly (2), also I have check with the firewall log seem there is a cert get expired (3), seem the cert not get install correctly (4)?

1.

2.


3. but some how in my firewall log could see that it was connected to pkg.origin.jenkins.io 

4. seem all of then also without a valid cert path...

call using http

[jenkins-infra-bot]

ianw:

IdenTrust DST Root CA X3 Expiration (September 2021). It's bundled with the JDK.

Please be aware that the "IdenTrust DST Root CA X3" root expiring on 9/30/2021 has been replaced with the "IdenTrust Commercial Root CA 1" self-signed root which is also trusted by the major browsers and root stores since 1/16/2014. You may download the IdenTrust Commercial Root CA 1 at this link: Root Certificate Download.

If you have appliances that are not dynamically updating the root trust chain, they need to be manually updated with the self-signed "IdenTrust Commercial Root CA 1" which can be downloaded at this link: Root Certificate Download.

[jenkins-infra-bot]

JIRAUSER133977:

Hi Ian Williams, ref to https://bugs.openjdk.java.net/browse/JDK-8161008, I have replaced the latest cert into jdk default keystore and jenkins customs keystore as well but unlucky not work. can I ask a stupid question as I have blocked in here for a whiles haha.. , is that the issue located on java level instead of other side?  

[jenkins-infra-bot]

dduportal:

Hello Karlos, you might want to switch the discussion on community.jenkins.io as the JIRA issues for the "INFRA" project are aimed at issues related to the Jenkins infrastructure.

As explained by other members in this thread (thanks to you all!), the Let's Encrypt legacy certificate trust chain named "DST Root CA X3" expired end of September 2021. You can have details on the Let's Encrypt blog post here: https://letsencrypt.org/2021/10/01/cert-chaining-help.html (and on the numerous other resources already posted).

Why are you impacted? The reason is that the Jenkins infrastructure is using Let's Encrypt as a certificate provider, and ALL certificate emitted by Let's encrypt are signed by their own trust authority. During the past years, both the legacy and the new one were usable for these emitted certificate. Only the new one can be used, and there is nothing we can do about this as it is a normal, classic behavior in the world of security practises (e.g. rotating machine to machine credentials as a regular procedure).

85% of people facing this issue should upgrade their environment (the "ca-certificate" package on Linux distribution, the OpenJDK used by Jenkins, or the Docker image used to run Jenkins, etc.) to solve this issue.
For the 15% others, our dear Ian Williaws gave the instruction earlier in this thread: TL;DR; if you cannot upgrade you packages/images, then you have to retrieve the new authority certificat chain and add it to your trust stores.

Thanks for your understanding, and do not hesitate to open a discussion on community.jenkins.io to get a broader community help (as JIRA issue might not get you a lot of visibility as the scope is pretty narrow).

[jenkins-infra-bot]

dduportal:

Hell Gabe Carnell thanks for reporting.

I was able to reproduce with the latest "stock" Ubuntu 20.04 official AMI on AWS EC2: I confirm that the (viable on sustainable) fix is to upgrade the ca-certificate packages (as a general sane practise, upgrading regularly your OS is recommended).

This situation will exists until Canonical updates the official Ubuntu 20.04 AMI (with recent packages).

Sorry for the inconvenience: we (infrasturcture Jenkins team) were caught off-guard by this as well.

[jenkins-infra-bot]

dduportal:

Closing the issue as there is nothing we can do on the infrastructure side.

[jenkins-infra-bot]

[Duplicates: INFRA-3105