RHEL/7.2 setting repo_gpgcheck=1 causes yum update to fail

[openprivacy]

I believe this worked in RHEL/7.1 but now having repo_gpgcheck=1 in /etc/yum.conf is failing on our AWS instances with e.g., http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/repomd.xml.asc: [Errno 14] HTTP Error 404 - Not Found

This bug is reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1360939

My question is: is this a bug in RHEL or should the SSG no longer test for this value?

[shawndwells]

Valid check. Reflects a bug to Red Hat corporate though.

repo_gpgcheck will evaluate the signatures of the yum repo metadata. Red Hat stopped publishing those, so the check will fail for redhat.com repos. It's a totally valid check though. Ensures those who create their own local yum repositories sign their content.

The BZ you cited calls for Red Hat to start publishing gpg content for repo metadata. BugZillas do not carry SLA's though -- customer support cases do. To help move Red Hat forward, signing repo data in the future, we need customer cases.

If this is important to you (broad community, not just @openprivacy ): Please open a Red Hat support case and send me the ticket number (shawn@redhat.com). I will administratively associate your ticket with the bugzilla -- effectively giving us means to document customer demand to have Red Hat start publishing signatures for repo data.

In the mean time, I'll make a patch to the XCCDF indicating this is a known issue with Red Hat repos.

[openprivacy]

https://access.redhat.com/support/cases/#/case/01752320

[shawndwells]

On 12/5/16 1:25 PM, Fen Labalme wrote:

https://access.redhat.com/support/cases/#/case/01752320

Thanks Fen! I've associated the ticket with bugzilla.

Also made an internal note on your ticket. Should stop the request for sosreport's ;)

-- Shawn Wells Chief Security Strategist U.S. Public Sector shawn@redhat.com | 443.534.0130

[shawndwells]

update:

If you are interested in Red Hat providing gpg signed repo data then you need to follow the content delivery ticket

https://projects.engineering.redhat.com/browse/DELIVERY-2451

If you are interested in Satellite 6 functionality then you want to follow:

https://bugzilla.redhat.com/show_bug.cgi?id=1410638

for the capability to sign the repositories satellite serves.

[openprivacy]

On Mon, Feb 27, 2017 at 2:58 PM, Shawn Wells notifications@github.com wrote:

update:

If you are interested in Red Hat providing gpg signed repo data then you need to follow the content delivery ticket

https://projects.engineering.redhat.com/browse/DELIVERY-2451

That link didn't work for me, but this one does (RHEL/7 issue): https://access.redhat.com/support/cases/#/case/01752320

If you are interested in Satellite 6 functionality then you want to follow:

https://bugzilla.redhat.com/show_bug.cgi?id=1410638

for the capability to sign the repositories satellite serves.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/1596#issuecomment-282834379, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJlp5ScfcZr1ZOGlc3Et14iHIXsPEVAks5rgyrZgaJpZM4K_e8K .

[jmnielsen7]

@shawndwells I hope this is not too far out of scope, but I thought I'd try my luck here. After hardening a system with the latest RHEL 7 SSG datastream (1.33, last I checked) somehow by default now any repos we add expect to have the metadata signed (although 'man yum.conf' apparently indicates differently by saying that the default setting is repo_gpgcheck=0 if not explicitly set otherwise - so I'm not sure what is overriding that). Since for third party repos that we add it will complain about the absence of repomd.xml.asc on each of them now, we've had to resort to setting gpgcheck=1 but repo_gpgcheck=0 on our repos for the time being to install packages (but worry this will produce a STIG non-compliance finding).

A concrete example is for Jenkins. Their repo usage instructions for RHEL are very simple (3 steps):

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins

Source: https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Red+Hat+distributions

Yet, that key seems to only be used for validating the RPMs themselves (gpgcheck) and not the repo metadata (repo_gpgpcheck), so we have to set the latter to 0 to avoid the repomd.xml.asc not found error. Taken on analogy to your above response about the situation with official Redhat repos it sounds like the repo maintainer has to publish a specific GPG key for validating the metadata. To be clear, would that be published as a separate (second) key that has to be imported and not a reuse of the GPG key used for validating the RPMs?

Having RHEL check for signed metadata is new to me, so I'm not really sure if (taking my specific example) whether that one GPG key (jenkins-ci.org.key) should have been enough to validate both the RPMs and the metadata or if it is typical to publish a second GPG key just for metadata. Any insights there?

[redhatrises]

somehow by default now any repos we add expect to have the metadata signed

That is correct. SSG is now setting repo_gpgcheck=1.

Having RHEL check for signed metadata is new to me, so I'm not really sure if (taking my specific example) whether that one GPG key (jenkins-ci.org.key) should have been enough to validate both the RPMs and the metadata or if it is typical to publish a second GPG key just for metadata. Any insights there?

Should only require the one gpg key. If you go here: https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/, you can see that repomd.xml.asc does not exist. If repomd.xml.asc exists, the repo was signed with a GPG key much like an RPM is signed with a GPG key.

[xmtrcv]

We ran into the same issue, finding out when attempting to yum install from a repo created from the RHEL 7.3 DVD source tree. Let me look through my documentation and update with some examples and hopefully a potential path forward.

[jmnielsen7]

@redhatrises

That is correct. SSG is now setting repo_gpgcheck=1.

Ah, I see it set in /etc/yum.conf now. I should have thought to look for a global setting there earlier.

Should only require the one gpg key. If you go here: https://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/, you can see that repomd.xml.asc does not exist. If repomd.xml.asc exists, the repo was signed with a GPG key much like an RPM is signed with a GPG key.

Okay, so basically regardless of whether the repo maintainer has signed their metadata/released repomd.xml.asc it would only require one key in either scenario. So most likely in my example with Jenkins since I cannot curl a repomd.xml.asc, and if I disable repo_gpgcheck on it it still validates signed RPMs (since I imported the key provided), then most likely Jenkins has just not signed their metadata. Boy, this is going to be fun...

[jan-cerny]

@redhatrises @shawndwells I run into this issue again today when I run remediation script for DISA STIG profile on RHEL7. Setting repo_gpgcheck to 1 breaks the the whole remediation for this profile, because then yum can't install the packages that the DISA STIG profile requires to be installed.

I think we should remove the remediation script for this rule. Or at least disable the remediation script temporarily, till Red Hat signes its repositories. Now it just breaks yum and doesn't improve security. Moreover, after this "remediation" is done, you can't install security updates, so the system is left vulnerable. And from user's perspective, the error message and recommendation that yum throws is misleading, because you as an user can't figure out that "repo_gpgcheck=1" caused the problem.

[xmtrcv]

@jan-cerny I can completely sympathize. The system undergoing remediation requires a signed repo in order to install additional packages, however, there is no failure that is obviously indicated. In our case, it was attempting to install 'esc' as required by remediation and failing to do so. I have since added that package into the kickstart, but a number of systems were built and remediated without having the package installed and will need to be patched.

Initially, I was setting repo_gpgcheck=0 at the end of my remediation script (package installation failures would have already occurred), but moving forward, we decided to sign the repo used as source to build/patch systems (since we self-sign some other provided repos with custom built packages.)

[shawndwells]

On 7/25/17 8:01 AM, Jan Černý wrote:

@redhatrises https://github.com/redhatrises @shawndwells https://github.com/shawndwells I run into this issue again today when I run remediation script for DISA STIG profile on RHEL7. Setting repo_gpgcheck to 1 breaks the the whole remediation for this profile, because then yum can't install the packages that the DISA STIG profile requires to be installed.

gpg enablement of Red Hat repos is being tracked downstream here: https://bugzilla.redhat.com/show_bug.cgi?id=1410638

Even more humorously, someone identified the algorithms used to gpg sign YUM repos break when the system is in FIPS mode, which makes it impossible to satisfy this control: https://access.redhat.com/solutions/2130631

It's apparently scheduled to be fixed in RHEL 7.4: https://bugzilla.redhat.com/show_bug.cgi?id=1220769

[For some reason this BZ is marked private to Red Hatters and partners. I do not know why. Link may not work for some people because of this]

So the current state of the union:

• Nobody can sign their repos because the gpg tooling is broke
• The tooling is scheduled to be fixed in RHEL 7.4
• Once RHEL 7.4 releases, Red Hat's entire build system must be updated to sign repos. TBD on schedule for that.
• Supplementary technologies, like RHN Satellite, also need to be updated. TBD schedule for that.

In the mean time, this leaves users screwed.

I think we should remove the remediation script for this rule. Or at least disable the remediation script temporarily, till Red Hat signes its repositories. Now it just breaks yum and doesn't improve security. Moreover, after this "remediation" is done, you can't install security updates, so the system is left vulnerable. And from user's perspective, the error message and recommendation that |yum| throws is misleading, because you as an user can't figure out that "repo_gpgcheck=1" caused the problem.

We can disable it in profiles that we control, e.g. OSPP... but still needs to be enabled in STIG (which aligns to DISA content).

@tbrunell: Maybe we can get DISA to mark this as a permanent finding and drop from STIG?

[jan-cerny]

@shawndwells We discussed this with @mpreisler . We suggested I will comment out the line of remediation that breaks the system and add a comment there or warning explaining why this is commented out and why it can't be remediated. I will provide a pull request.

[jan-cerny]

https://github.com/OpenSCAP/scap-security-guide/pull/2189

[jmnielsen7]

@shawndwells

Thanks for the detailed update above. As you suggested, that bugzilla link is internal to Redhat Developers and I cannot view it even when logged in. Are there any recent remarks there indicating that this has been addressed?

From what I can tell RHEL 7.4 has been released now though I haven't attempted to update any of our systems to it. Do you know of any updates on the status of signed repos and the RHN satellite with 7.4?

Thanks in advance.

[shawndwells]

On 10/9/17 10:46 AM, jmnielsen7 wrote:

@shawndwells https://github.com/shawndwells

Thanks for the detailed update above. As you suggested, that bugzilla link is internal to Redhat Developers and I cannot view it even when logged in. Are there any recent remarks there indicating that this has been addressed?

From what I can tell with a quick google search, RHEL 7.4 has been released now. Do you know of any updates on the status of signed repos and the RHN satellite?

Thanks in advance.

From a tooling perspective, libgcrypt tools were updated in RHEL 7.4, enabling the needed gpg crypto. Was released via: https://access.redhat.com/errata/RHBA-2017:2006

From a Satellite enablement perspective, that is being tracked in the public BugZilla here: https://bugzilla.redhat.com/show_bug.cgi?id=1410638

I'm just returning from a week of PTO and may have missed updates. Asked for one directly in the bugzilla (since it's public, you can subscribe to updates against it).

[jmnielsen7]

@shawndwells Excellent. Thank you. So with this libgcrypt update though does that mean that Redhat has already signed all their (global/non-satellite) RHEL 7.4 package repos with the requisite .asc files to make this work and satisfy DISA STIG requirements, or is that still TBD?

I know you mentioned the issue with this not working on a host OS in previous releases when FIPS mode was enabled, but that only seems to affect the end user's OS in so far as they are signing their own custom repos, correct? Currently I'm just wondering if the official RHEL repos for 7.4 are themselves signed now.

[agilmore2]

https://access.redhat.com/support/cases/#/case/01961866

Also commented on bugzilla.

Thanks for keeing track of this!

[openprivacy]

AFAICT, repos metadata is still not being published: https://access.redhat.com/support/cases/#/case/01752320

=Fen

On Fri, Oct 27, 2017 at 12:07 PM, agilmore2 notifications@github.com wrote:

https://access.redhat.com/support/cases/#/case/01961866

Also commented on bugzilla.

Thanks for keeing track of this!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/1596#issuecomment-340014780, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJlpzOrejqn01_JydMp8algZyqdTeAYks5swf-0gaJpZM4K_e8K .

[jmnielsen7]

@agilmore2 @openprivacy For some reason I can't see either of those cases when I click on the link (while logged in, of course). I get an ascii art "404" printed to the screen followed by the message: "Not Found. The page you are looking for is not here. It might have been moved, removed, or had its name and address changed. It might otherwise be temporarily unavailable for technical reasons."

[agilmore2]

These are open support cases. As a Red Hat customer, those cases are generally private. I provided the link to mine to allow @RH folks to link the cases to the bugzilla. My support case essentially said, "I need this. Link to bugzilla please."

On Tue, Oct 31, 2017 at 9:09 AM, jmnielsen7 notifications@github.com wrote:

@agilmore2 https://github.com/agilmore2 @openprivacy https://github.com/openprivacy For some reason I can't see either of those cases when I click on the link (while logged in, of course). I get an ascii art "404" printed to the screen followed by the message: "Not Found. The page you are looking for is not here. It might have been moved, removed, or had its name and address changed. It might otherwise be temporarily unavailable for technical reasons."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/1596#issuecomment-340792887, or mute the thread https://github.com/notifications/unsubscribe-auth/AGiOyvqQV-nDVV-6MHaJAJOwyR5DSILqks5sxzgYgaJpZM4K_e8K .

[shawndwells]

@agilmore2 appreciate the case link! I see it's been added as a customer request for that bugzilla (which is awesome, because there are tons of them now).

As indicated in the public bugzilla, patches were merged upstream a weekish ago:

This has been merged into upstream Pulp.  Relevant commits are:
https://github.com/pulp/pulp_rpm/commit/f73805f626d96596f9ae962ab6d787d2e001f02c
https://github.com/pulp/pulp_rpm/commit/5393773e361ed548b72696652f338b76908429d0
https://github.com/pulp/pulp_rpm/commit/b3e2dd8bd8c43abb4144cc9d7a121fec5e5f5899

With the exception of the documentation changes (which can be excluded), these commits apply cleanly to the Pulp v2.8.7.18 that is used by Satellite v6.2.12.

AS JUST A PERSONAL OPINION it looks like this will be enabled in future satellite versions and the conversation is now about backporting to existing Satellite codebase. Definitely keep an eye on the bugzilla for updates.

[shawndwells]

Oh snap. Just noticed the pm_ack flag was set on the bugzilla. Indicates product management has approved this to go into the next Satellite release.

Product Management and QE are two of many stakeholders.... still needs development approval (e.g. can the patches be merged successfully?). Safe to say this issue is progressing smoothly, but not far enough along to be a feature commitment yet. Keep the customer requests coming!

[PaulSD]

I have posted backported patches and instructions for enabling this on Satellite 6.2.12 here: https://bugzilla.redhat.com/show_bug.cgi?id=1410638#c17

[openprivacy]

https://access.redhat.com/support/cases/#/case/01752320 - slow... deserves to be backlogged...

From: Janorkar, Anuja on Sep 19 2018 at 05:52 PM -07:00

Unfortunately, we have not received the update on this. We will get back to you as soon as we get an update. We appreciate your patience.

Best Regards, Anuja J. Global Support Services, Red Hat

[redhatrises]

Closing this upstream ticket as this now is an issue with the products and not the content. Please keep all downstream tickets open to resolve this issue. Thanks.