jlebon
commented
6 years ago Hmm, so something dropped it as a dep? I suppose we could manually add it back in the manifest, though it'd be nice to track down what dropped the dep and why.
Open miabbott opened 6 years ago
jlebon
commented
6 years ago Hmm, so something dropped it as a dep? I suppose we could manually add it back in the manifest, though it'd be nice to track down what dropped the dep and why.
miabbott
commented
6 years ago I mean, this is a testing stream...i'm not opposed to the hammer approach.
cgwalters
commented
6 years ago Without actually investigating my offhand guess is that something in CentOS chagned to explicitly neuter that package in 7.5.
miabbott
commented
6 years ago This is still an issue in the latest CAHC builds...
I tried to dig through the dependencies and what not:
# docker run -it --rm registry.centos.org/centos repoquery --whatprovides /etc/rhsm/ca/redhat-uep.pem
python-rhsm-certificates-0:1.19.10-1.el7_4.x86_64
# docker run -it --rm registry.centos.org/centos repoquery --whatrequires python-rhsm-certificates
python-rhsm-0:1.19.10-1.el7_4.x86_64
# docker run -it --rm registry.centos.org/centos repoquery --whatprovides python-rhsm
subscription-manager-rhsm-0:1.20.11-1.el7.centos.x86_64
python-rhsm-0:1.19.10-1.el7_4.x86_64
# rpm -qa | grep rhsm
subscription-manager-rhsm-certificates-1.20.11-1.el7.centos.x86_64
subscription-manager-rhsm-1.20.11-1.el7.centos.x86_64The spec file for subscription-manager seems to indicate that the redhat-uep.pem cert should be installed by subscription-manager-rhsm-certificates:
So....kind of back to where we started. ¯_(ツ)_/¯
Seems to support what @cgwalters said about the certs getting neutered. Not sure where to look for evidence of that.
miabbott
commented
6 years ago This appears to affect all of CentOS
https://lists.centos.org/pipermail/centos-devel/2018-June/016749.html
brianUK007
commented
5 years ago @miabbott I just built a 3.10 OKD cluster on centos7 with ansible and containerized gluster for dynamic storage. I had run a git checkout release-3.10 for https://github.com/openshift/openshift-ansible.git in late october / november to build the cluster. Upgrading with Gluster is a bit more difficult than just running the upgrade script.
Is there a workaround to this? Could I inject the cert into the worker nodes to get around this issue? I've wanted to demo the .Net examples to some of our dev team working with .Net but running into issues pulling images from RedHat on OKD 3.10.
miabbott
commented
5 years ago @brianUK007 I decided to have another look at this to see what I can find. The idea that the CA certs are getting neutered as part of the build process appears to be confirmed. I believe I found the proper spec file which shows an explicit removal of the certs on CentOS:
https://git.centos.org/blob/rpms!subscription-manager.git/c7/SPECS!subscription-manager.spec#L562
But, the registry still has to offer its cert when negotiating SSL, so we can grab it and stick it where it needs to be.
Using some help from StackOverflow, I grabbed the cert and stuck it in the right place for docker:
# docker pull registry.access.redhat.com/rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ...
open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory
# openssl s_client -showcerts -servername registry.access.redhat.com -connect registry.access.redhat.com:443 </dev/null 2>/dev/null | openssl x509 -text > /etc/rhsm/ca/redhat-uep.pem
# docker pull registry.access.redhat.com/rhel7/openscap
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscap ...
latest: Pulling from registry.access.redhat.com/rhel7/openscap
9a1bea865f79: Pull complete
602125c154e3: Pull complete
4f39a853bed4: Pull complete
20c68cea93f0: Pull complete
Digest: sha256:aa5ddb23af242da108ee0cfe227a96ced06ad398e4c8bb201aa837ca2837e432
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latestI'm not sure what redhat-entitlement-authority.pem is needed for (or where to get it), but this at least should get to the point where you can pull images with docker.
brianUK007
commented
5 years ago Thanks Micah
THIS E-MAIL MAY CONTAIN CONFIDENTIAL AND/OR PRIVILEGED INFORMATION. IF YOU ARE NOT THE INTENDED RECIPIENT (OR HAVE RECEIVED THIS E-MAIL IN ERROR) PLEASE NOTIFY THE SENDER IMMEDIATELY AND DESTROY THIS E-MAIL. ANY UNAUTHORISED COPYING, DISCLOSURE OR DISTRIBUTION OF THE MATERIAL IN THIS E-MAIL IS STRICTLY FORBIDDEN.
IN ACCORDANCE WITH MIFID II RULES ON INDUCEMENTS, THE FIRM'S EMPLOYEES MAY ATTEND CORPORATE ACCESS EVENTS (DEFINED IN THE FCA HANDBOOK AS "THE SERVICE OF ARRANGING OR BRINGING ABOUT CONTACT BETWEEN AN INVESTMENT MANAGER AND AN ISSUER OR POTENTIAL ISSUER"). DURING SUCH MEETINGS, THE FIRM'S EMPLOYEES MAY ON NO ACCOUNT BE IN RECEIPT OF INSIDE INFORMATION (AS DESCRIBED IN ARTICLE 7 OF THE MARKET ABUSE REGULATION (EU) NO 596/2014). (https://www.handbook.fca.org.uk/handbook/glossary/G3532m.html) COMPANIES WHO DISCLOSE INSIDE INFORMATION ARE IN BREACH OF REGULATION AND MUST IMMEDIATELY AND CLEARLY NOTIFY ALL ATTENDEES. FOR INFORMATION ON THE FIRM'S POLICY IN RELATION TO ITS PARTICIPATION IN MARKET SOUNDINGS, PLEASE SEE https://www.horizon-asset.co.uk/market-soundings/.
HORIZON ASSET LLP IS AUTHORISED AND REGULATED BY THE FINANCIAL CONDUCT AUTHORITY.
From: Micah Abbott [mailto:notifications@github.com] Sent: Wednesday, November 21, 2018 3:05 PM To: CentOS/sig-atomic-buildscripts Cc: Brian Saltzman; Mention Subject: Re: [CentOS/sig-atomic-buildscripts] CAHC missing CA cert for Red Hat (#329)
@brianUK007https://github.com/brianUK007 I decided to have another look at this to see what I can find. The idea that the CA certs are getting neutered as part of the build process appears to be confirmed. I believe I found the proper spec file which shows an explicit removal of the certs on CentOS:
https://git.centos.org/blob/rpms!subscription-manager.git/c7/SPECS!subscription-manager.spec#L562https://git.centos.org/blob/rpms!subscription-manager.git/c7/SPECS!subscription-manager.spec#L562
But, the registry still has to offer its cert when negotiating SSL, so we can grab it and stick it where it needs to be.
Using some help from StackOverflowhttps://stackoverflow.com/a/35442314, I grabbed the cert and stuck it in the right place for docker:
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscaphttp://registry.access.redhat.com/rhel7/openscap ...
open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/openscaphttp://registry.access.redhat.com/rhel7/openscap ...
latest: Pulling from registry.access.redhat.com/rhel7/openscaphttp://registry.access.redhat.com/rhel7/openscap
9a1bea865f79: Pull complete
602125c154e3: Pull complete
4f39a853bed4: Pull complete
20c68cea93f0: Pull complete
Digest: sha256:aa5ddb23af242da108ee0cfe227a96ced06ad398e4c8bb201aa837ca2837e432
Status: Downloaded newer image for registry.access.redhat.com/rhel7/openscap:latesthttp://registry.access.redhat.com/rhel7/openscap:latest
I'm not sure what redhat-entitlement-authority.pem is needed for (or where to get it), but this at least should get to the point where you can pull images with docker.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/CentOS/sig-atomic-buildscripts/issues/329#issuecomment-440695888, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AgsNM2vf5aoH3cxCzez5_Yhpl7ZWd1NZks5uxWuPgaJpZM4T-WrU.
The sanity tests on CAHC have been failing for an embarrassing amount of time, but I just did some digging after @jlebon asked about it.
The root cause looks like a missing CA cert for Red Hat things. This was observed when doing a
docker pullfrom the registry:The
redhat-ca.crtfile is usually a symlink to/etc/rhsm/ca/redhat-uep.pem. On the CAHC stream, this was provided bypython-rhsm-certificates, but recent composes have caused this package to drop out of the compose. (FWIW, on RHELAH the cert is provided bysubscription-manager-rhsm-certificates, but this package is basically empty in CentOS land)The first compose where it appears this package was removed was on April 26, commit
4d12023435213f8c639337679d3f093f0188cfe8eaf77f4d5963ba5e35aea7e7