PAM Login / PKCS library issue after applying 1.33 datastream

[jmnielsen7]

Edit: Long story short, I forgot to install the pam_pkcs11 package before running the SSG remediation and that caused this issue.

This issue may be way off in the outfield but I thought I'd try my luck reporting it here. We are rolling out new RHEL 7.3 images which we have injected a kickstart into to take advantage of the oscap addon like so:

%addon org_fedora_oscap
    content-type = scap-security-guide
    profile = stig-rhel7-workstation-upstream
%end

Long story short, this part works fine and it is hardened up to certain percentage out of the box now, but we have been afterward coming to the GitHub site here to grab the latest data stream to manually apply it so that we stay up to date with the newest releases. With the latest 1.33 release (scap-security-guide-0.1.33.zip) once we apply the RHEL 7 DISA STIG profile (full command: _sudo oscap xccdf eval --remediate --profile xccdf_org.ssgproject.content_profilestig-rhel7-disa --results ~/scan-xccdf-results.xml ./ssg-rhel7-ds.xml) and then reboot we are getting the dreaded "Login incorrect" issue where as soon as you type a username and hit enter it displays that message ("Login incorrect") before (and never reaching) a password prompt. The /var/log/secure log shows the problem:

PAM unable to dlopen(/usr/lib64/security/pam_pkcs11.so): /usr/lib64/security/pam_pkcs11.so: cannot open shared object file: No such file or directory

And sure enough that file is missing. Manually (re)installing the pam_pkcs11 rpm that provides that library file will then enable users to log in again, but for some reason it leaves an ugly warning on the login screen nonetheless (ERROR: pam_pkcs11.c:318: load_pkcs11_module() failed). I basically need to find out what removed that pam_pkcs11.so file. It is similar to this person's problem, except the suggested solution (internet/networking connection) was not an issue for us: https://www.centos.org/forums/viewtopic.php?t=61314

What we are going to test next is a fresh RHEL 7.3 deploy without the oscap addon in the kickstart which was pre-hardening it with the "stig-rhel7-workstation-upstream" profile, and instead just grab the latest datastream 1.33 after a standard install and apply the "stig-rhel7-disa" profile in case there is some weird issue with it being due to the cumulative application of oscap profiles.

Any suggestions would be welcome, or a pointer of a more general discussion area where I could post this question.

[xmtrcv]

I currently provision guest images and hardware installs in a standalone network, using kickstart with oscap eval/remediation to occur on first boot (after kickstart install). I've have not experienced any issues with "Login incorrect" or PAM errors (even as the result of no outside networking as the CentOS thread would suggest). I am using ssg 0.1.33 with the "stig-rhel7-disa" profile also.

There are two different profile names you are referencing: addon vs. manual, is that intentional?

On Tue, May 30, 2017 at 10:29 AM, jmnielsen7 notifications@github.com wrote:

This issue may be way off in the outfield but I thought I'd try my luck reporting it here. We are rolling out new RHEL 7.3 images which we have injected kickstart into to take advantage of the oscap addon like so:

%addon org_fedora_oscap content-type = scap-security-guide profile = stig-rhel7-workstation-upstream %end

Long story short, this part works fine and it is hardened up to certain percentage out of the box now, but we have been coming to the GitHub site here to grab the latest data stream to manually apply it. With the latest 1.33 release (scap-security-guide-0.1.33.zip) once we apply the RHEL 7 DISA STIG profile (full command: sudo oscap xccdf eval --remediate --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa --results ~/scan-xccdf-results.xml ./ssg-rhel7-ds.xml) and then reboot we are getting the dreaded "Login incorrect" issue where as soon as you type a username and hit enter it displays that message before a password prompt. The /var/log/secure log shows the problem:

PAM unable to dlopen(/usr/lib64/security/pam_pkcs11.so): /usr/lib64/security/pam_pkcs11.so: cannot open shared object file: No such file or directory

It is similar to this person's problem, except the suggested solution (internet/networking connection) was not an issue for us: https://www.centos.org/forums/viewtopic.php?t=61314

What we are going to test next is a fresh RHEL 7.3 deploy without the oscap addon in the kickstart which was pre-hardening it with the "stig-rhel7-workstation-upstream" profile, and instead just grab the latest datastream 1.33 after a standard install and apply the "stig-rhel7-disa" profile in case there is some weird issue with it being due to the cumulative application of oscap profiles.

Any suggestions would be welcome, or a pointer of a more general discussion area where I could post this question.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/2024, or mute the thread https://github.com/notifications/unsubscribe-auth/ARj4MrFQl8tbu1w13x5JPWsFQpD66nWwks5r_CfkgaJpZM4NqYKU .

[jmnielsen7]

Yes, so the deal with the profile names is that we are using "stig-rhel7-workstation-upstream" as the profile name in the kickstart which is one of the predefined profiles that can be used as one of the "org_fedora_oscap" addon options/parameters I assume (this kickstart was supplied to me as-is by someone else I work with so I'm taking the addon options for granted). I don't know what command the plugin runs in the background/automatedly to actually apply it - if that's even relevant.

Then subsequently, after the automated install, when I go to download (wget) the latest data stream, unzip it, and then I list the profiles in the ssg-rhel7-ds.xml, however, I don't see any identically named profile:

sudo oscap info "./ssg-rhel7-ds.xml"
    Document type: Source Data Stream
    Imported: 2017-04-29T04:58:51

Stream: scap_org.open-scap_datastream_from_xccdf_ssg-rhel7-xccdf-1.2.xml
    Generated: (null)
    Version: 1.2
    Checklists:
        Ref-Id: scap_org.open-scap_cref_ssg-rhel7-xccdf-1.2.xml
            Status: draft
            Generated: 2017-04-29
            Resolved: true
            Profiles:
                xccdf_org.ssgproject.content_profile_standard
                xccdf_org.ssgproject.content_profile_pci-dss
                xccdf_org.ssgproject.content_profile_C2S
                xccdf_org.ssgproject.content_profile_rht-ccp
                xccdf_org.ssgproject.content_profile_common
                xccdf_org.ssgproject.content_profile_stig-rhel7-disa
                xccdf_org.ssgproject.content_profile_stig-rhevh-upstream
                xccdf_org.ssgproject.content_profile_ospp-rhel7
                xccdf_org.ssgproject.content_profile_cjis-rhel7-server
                xccdf_org.ssgproject.content_profile_docker-host
                xccdf_org.ssgproject.content_profile_nist-800-171-cui

If there were an identically named one I would have chosen it (and the only other one with "upstream" has "rhevh" in it, and I don't know what that signifies). Since we actually follow the DISA STIG hardening guidelines at my workplace I decided to go with the "stig-rhel7-disa" which seemed closest to our needs. So then I just manually run the command I provided before.

Do you see a problem with applying two different profiles like that? I admittedly need to do a little more homework on the profile options for the addon that I can use in the kickstart (like if there is a DISA one there as well), but I was supplied the kickstart as-is. What ought I to change in this procedure if anything? Or should that actually work with no problems, at least conceptually?

[jmnielsen7]

I wonder if it has to do with the addon referring to "fedora" and later from the data stream I download I'm using ssg-rhel7-ds.xml instead of ssg-fedora-ds.xml (which is in the same release zip file). But I should reiterate that the configuration of the existing oscap portion of the kickstart creates a perfectly working RHEL 7.3 instance that is hardened. It's only after I apply the data stream from here that it causes problems: https://github.com/OpenSCAP/scap-security-guide/releases/download/v0.1.33/scap-security-guide-0.1.33.zip

[xmtrcv]

I think the %addon syntax is correct, when referencing https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/sect-kickstart-syntax.html

Is it possible to use:

%addon org_fedora_oscap content-type = scap-security-guide profile = stig-rhel7-disa %end

in the initial kickstart, rather than remediating with one profile, and then remediating again with another profile?

Though, I am not sure what version of the SSG the OSCAP Anaconda section is referencing / pulling down during execution. (I am assuming <0.1.33)

I was always under the assumption that remediation on a system would be done with one profile only. There might be conflicts, so YMMV.

Does the system to the same percentage or better if you remove the OSCAP Anaconda %addon and perform the remediation (using 0.1.33 + stig-rhel7-disa) after kickstart/imaging? That might be the better solution if you are looking to remediate against the latest SSG release (0.1.33). (FWIW, the aforementioned is the configuration/setup I use.)

As a side note: 'rhvh' = "Red Hat Virtualization Hypervisor"

On Tue, May 30, 2017 at 7:43 PM, jmnielsen7 notifications@github.com wrote:

I wonder if it has to do with the addon referring to "fedora" and I'm using ssg-rhel7-ds.xml instead of ssg-fedora-ds.xml (which is in the same release zip file). But I should reiterate that the configuration of the existing oscap portion of the kickstart creates a perfectly working RHEL 7.3 instance that is hardened. It's only after I apply the data stream here that is causes problems: https://github.com/OpenSCAP/scap-security-guide/releases/ download/v0.1.33/scap-security-guide-0.1.33.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/2024#issuecomment-305039935, or mute the thread https://github.com/notifications/unsubscribe-auth/ARj4MoXB3gZI9oMrwi9v2a-1iBovH5v8ks5r_KmlgaJpZM4NqYKU .

[jmnielsen7]

Okay, so I just learned that the backstory to this was that the DISA STIG profile not long ago was still in draft mode and that is the only version available through the plugin currently (so I've heard), or at least that is the case with the xml files in the scap security guide rpm package that comes with whatever release of RHEL 7.3 we burned to a DVD. When the disa stig profile was applied it still left 60 or so non-STIG-compliant findings, whereas applying the stig-rhel7-workstation-upstream profile left only 7 or 8 findings. So it made more sense to use the latter to bootstrap us, since we are deploying this to multiple systems.

How is this being supported between major releases of RHEL? Basically until RHEL 7.4 stable is rolled out with the latest scap security guide updates it seems likely that many people who burned a plain RHEL 7.3 ISO to disk will have to deal with the draft version of the DISA STIG when deploying systems with kickstarts (whether locally injected on a DVD or deployed via PXE).

Also I think I heard that the draft profile name was different than the current upstream profiles or something like that, so that would make it difficult for admins to anticipate profile name changes to ensure forward compatibility when applying the latest data streams on top of what they deployed a system with previously. I'm not sure how far back that comment I heard applies though. Will that become more consistent in the future? Are you a contributor on this project/in a position to answer that? If not can we tag anyone who is?

My boss was talking to Shawn Wells at the recent Red Hat Summit not long ago about this. He seems to be the main RHEL representative on this project. I guess the issue is just finding a process by which we can easily use the latest RPMs available and then fill the gaps in non-STIG compliant findings afterward with a manual application of the latest data stream release. Is that how you are doing it in your environment, or do you not use anything from the distro RPMs at all?

Thanks for your help so far!

[xmtrcv]

I apologize, but I am not in a position to answer OpenSCAP or SCAP Security Guide roadmap-related questions. I can only detail what I currently use/trial/error/experience in my environment.

Currently, I provision VMs & hardware using kickstart to install (very) minimal RHEL 7.3. (specific application server)

Initial remediation handled by a script set to run (via kickstart %post) on first boot (which is after kickstart reboot) using:

• scap-security-guide: 0.1.33
• input file: ssg-rhel7-ds.xml
• profile: xccdf_org.ssgproject.content_profile_stig-rhel7-disa

Typically, this leaves approximately ~20 OSCAP fails/errors/false positives/etc. There are ~20 RHEL7 STIG Release: 1 (27 Feb 2017) findings not handled by OpenSCAP/SSG.

Outstanding deltas, whether they are oscap fails/errors, false positives, or missing findings are remediated and/or verified manually in the latter portion of the script.

Since the DISA RHEL7 STIG has been such a moving target, from draft to (finally) release, I would imagine the SSG will be too for a bit, so will have to supplement where it makes sense in the interim.

On Wed, May 31, 2017 at 10:05 AM, jmnielsen7 notifications@github.com wrote:

Okay, so I just learned that the backstory to this was that the DISA STIG profile not long ago was still in draft mode, or at least that is the case with the xml files that come with the scap security guide rpm package that comes with whatever release of RHEL 7.3 we burned to a DVD. When applied it still left 60 or so non-STIG compliant findings, whereas applying the stig-rhel7-workstation-upstream left only 7 or 8. So it made more sense to use the latter to bootstrap us.

How is this being supported between major releases of RHEL? Basically until RHEL 7.4 stable is rolled out with the latest scap security guide updates it seems likely that many people who burned a plain RHEL 7.3 ISO to disk will have to deal with the draft version of the DISA STIG when deploying systems with kickstarts (whether locally injected on a DVD or deployed via PXE). Are you a contributor on this project/in a position to answer that?

Thanks for your help so far.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenSCAP/scap-security-guide/issues/2024#issuecomment-305196941, or mute the thread https://github.com/notifications/unsubscribe-auth/ARj4MtgiEX1B8SnVdZz6Yw_76TFGPONPks5r_XOSgaJpZM4NqYKU .

[jmnielsen7]

Update:

Well, wow, I was able to reproduce the problem from a completely fresh install of RHEL 7.3 with no addons or other openscap configuration to start with (the oscap tool was not even installed after successful installation). The OS was installed from the fairly bland kickstart, however note the aide and libreswan packages:

%packages

@Base @Core

aide

libreswan

%end # End of %packages section

(Anything problematic there?)

The OS worked fine so I decided to then go grab the openscap release and apply it. I simply followed these steps (after yum installing the oscap tool), and now I cannot log in again ("Login incorrect") after applying the DISA STIG profile:

wget https://github.com/OpenSCAP/scap-security-guide/releases/download/v0.1.33/scap-security-guide-0.1.33.zip unzip scap-security-guide-0.1.33.zip cd scap-security-guide-0.1.33/ sudo oscap xccdf eval --remediate --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa --results ~/scan-xccdf-results.xml ./ssg-rhel7-ds.xml

I noticed this was in my /etc/pam.d/system-auth file:

auth [success=done authinfo_unavail=ignore ignore=ignore default=die] pam_pkcs11.so nodebug

But the pam_pkcs11.so file is missing and the package is not installed on the OS at all. So either:

1. Upon a fresh & unhardened install (now I'm going to have to do another install and check) the pam_pkcs11 package is installed and something adds the pam_pkcs11.so line to PAM files from the get go, and then afterwards running the DISA STIG oscap profile removes the package (but leaves the line in the PAM files). -OR- 2. Upon a fresh & unhardened install the pam_pkcs11 package is not installed, but when I run the DISA STIG oscap profile it does something that adds the pam_pkcs11.so line to PAM, thus causing a problem when the required library is missing.

Or it could be something else I haven't though of yet. Any thoughts here?

[jmnielsen7]

I have verified now that the scenario is (in keeping with scenario #2 above) that from a fresh install the pam_pkcs11 package is not installed nor are there any pam_pkcs11.so entries in any of the PAM files except for smartcard-auth. So that means that that something in the DISA STIG hardening profile is adding the entries with pam_pkcs11.so to the system-auth file, but is not installing the corresponding package, thus causing the issue.

I tried searching the "fix text" of various DISA STIG entries and couldn't find a reference to PKCS anywhere. Does anyone have an idea what might be doing this?

[xmtrcv]

A self-assessment of the system should be performed against Red Hat 7 STIG - Ver 1, Rel 1.

The self-assessment will help to establish the baseline for what is required to configure an OS installation (partitions, accounts/encryption, bootloader, SELinux, packages, etc.) with regards to security compliance.

From there, additional remediation can be done manually or OSCAP or a combination of both, via %post, %addon, or after installation is complete.

OSCAP+SSG is meant to supplement STIG remediation, but it will not automagically resolve for what packages are required when configuring for security compliance. It simply configures the system under the assumption the packages are installed per the STIG.

I apologize, as I mean to ask much earlier if pam_pkcs11 was installed as a package, but had just assumed it was per the STIG documentation (see below).

Rule Title: The operating system must have the required packages for multifactor authentication installed. STIG ID: RHEL-07-041001 Rule ID: SV-87041r2_rule Vuln ID: V-72417 Severity: CAT II Class: Unclass

Check Content: Verify the operating system has the packages required for multifactor authentication installed.

Check for the presence of the packages required to support multifactor authentication with the following commands:

# yum list installed esc 
esc-1.1.0-26.el7.noarch.rpm 

# yum list installed pam_pkcs11
pam_pkcs11-0.6.2-14.el7.noarch.rpm 

# yum list installed authconfig-gtk 
authconfig-gtk-6.1.12-19.el7.noarch.rpm

If the "esc", "pam_pkcs11", and "authconfig-gtk" packages are not installed, this is a finding.

(My systems do not have a GUI, so authconfig-gtk is not installed.)

tl;dr You have to install the pam_pkcs11 package as part of the kickstart.

Here's an example of STIG-mandated package installs (from my kickstart):

# aide                          V-71975,V-72063,V-72069,V-72071,V-72073
# dracut-fips                   V-72067
# ntp                           V-72269
# pam_pkcs11,coolkey            V-72417,V71965,V-27433
# policycoreutils-python        V-71971
# screen                        V-71987
#
aide                            # `- installs 001 package(s)
dracut-fips                     # `- installs 002 package(s)
ntp                             # `- installs 003 package(s)
pam_pkcs11                      # `- installs 002 package(s)
coolkey                         # `- installs 004 package(s)
policycoreutils-python          # `- installs 007 package(s)
screen                          # `- installs 001 package(s)
# `- installs 020 package(s): 020

[lmeinecke]

^ That's good info. Thanks for posting.

[jmnielsen7]

Ah, thank you very much @xmtrcv . I've been looking for a resource (other than a gui/thick client tool) to do quick searches on the fix text for RHEL 7 STIG entries, and your reference to RHEL-07-041001 led me to a google result that seems to do that (per severity level) here: http://rhel7stig.readthedocs.io/en/latest/medium.html

Also your kickstart entry is very helpful. Thanks again!

[jmnielsen7]

@xmtrcv Thank you very much for your help. That solved the problem. I just was not installing the required packages (in particular pam_pkcs11) before.

[mpreisler]

Reopening this because we should make sure that the package is installed in the bash and ansible remediations.

[redhatrises]

Closing. Validated that all current remediations contain command to install pam_pkcs11

[kenyon]

How was this actually fixed? I don't see any reference to a commit that would have fixed this.

On RHEL 7.5, doing a minimal kickstart install results in a system without the pam_pkcs11 package installed, preventing console login (ssh login works though). Red Hat bug report 1461330 and corresponding errata claims this was fixed in scap-security-guide-0.1.33-5.el7.noarch.rpm, but clearly it wasn't because RHEL 7.5 has scap-security-guide-0.1.36-7.el7 and this is still happening.

Example portion of minimal installation kickstart file:

%packages
@Base
%end
%addon org_fedora_oscap
    content-type = scap-security-guide
    profile = stig-rhel7-disa
%end

[alexhexabeam]

It is definitely still broken in RHEL 7.6. For anyone who hits this issue, you need to add BOTH coolkey and pam_pkcs11 to your packages list.

[redhatrises]

Not necessarily, both coolkey and pam_pkcs11 have been deprecated and will no longer appear in RHEL8+. Since RHEL 7.4+, opensc is the recommended smartcard utility over coolkey and pam_pkcs11.

[alexhexabeam]

@redhatrises Thank you for your response. That means this is definitely a bug in the remediation, since it still adds pam_pkcs11.so stuff to the pam config files.

I just confirmed that when installing opensc instead of coolkey and pam_pkcs11 in a kickstart of RHEL 7.6 with the remediation, I cannot log in.