openshift / compliance-operator

Operator providing OpenShift cluster compliance checks
Apache License 2.0
109 stars 110 forks source link

Scan results are not parsable #197

Closed jaedolph closed 4 years ago

jaedolph commented 4 years ago

Attempted to run a scan using the example procedure in the README.

I ran this in a development environment with 1 master and 2 worker nodes.

Seems like the scan ran with no reported error, and the scan reports as DONE. This is what the compliance suite CR looks like once the scan is done: (output of oc get compliancesuites example-compliancesuite -oyaml)

apiVersion: compliance.openshift.io/v1alpha1
kind: ComplianceSuite
metadata:
  creationTimestamp: "2020-03-30T06:27:31Z"
  generation: 1
  name: example-compliancesuite
  namespace: openshift-compliance
  resourceVersion: "93303"
  selfLink: /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/compliancesuites/example-compliancesuite
  uid: 0abe5e8f-c5c2-4a08-b1d6-c9bdbe8f9307
spec:
  autoApplyRemediations: false
  scans:
  - content: ssg-ocp4-ds.xml
    contentImage: quay.io/jhrozek/ocp4-openscap-content:latest
    name: workers-scan
    nodeSelector:
      node-role.kubernetes.io/worker: ""
    profile: xccdf_org.ssgproject.content_profile_moderate
  - content: ssg-ocp4-ds.xml
    contentImage: quay.io/jhrozek/ocp4-openscap-content:latest
    name: masters-scan
    nodeSelector:
      node-role.kubernetes.io/master: ""
    profile: xccdf_org.ssgproject.content_profile_moderate
status:
  remediationOverview:
  - apply: false
    remediationName: workers-scan-kernel-module-hfs-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-grub2-page-poison-argument
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-grub2-audit-argument
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-kernel-kexec-load-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv6-conf-all-accept-source-route
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-can-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-kernel-perf-event-paranoid
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-kernel-perf-event-paranoid
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-all-accept-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-icmp-ignore-bogus-error-responses
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-squashfs-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-cramfs-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-vfat-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-bluetooth-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-grub2-pti-argument
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-kernel-yama-ptrace-scope
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-icmp-echo-ignore-broadcasts
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv6-conf-default-accept-source-route
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-no-direct-root-logins
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-udf-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-grub2-audit-argument
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-atm-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-no-direct-root-logins
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-accept-redirects
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-send-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-no-empty-passwords
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-jffs2-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-kernel-unprivileged-bpf-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-tipc-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-jffs2-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-cramfs-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-kernel-yama-ptrace-scope
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-usb-storage-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-kernel-dmesg-restrict
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-sctp-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-grub2-page-poison-argument
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-kernel-unprivileged-bpf-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-kernel-dmesg-restrict
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-firewire-core-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-all-log-martians
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-atm-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-rp-filter
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-usb-storage-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-freevxfs-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-coredump-disable-storage
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-no-empty-passwords
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-accept-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-firewire-core-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-kernel-kexec-load-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-grub2-pti-argument
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-sctp-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-udf-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-icmp-echo-ignore-broadcasts
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-icmp-ignore-bogus-error-responses
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-send-redirects
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-tcp-syncookies
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-all-log-martians
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-accept-source-route
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv6-conf-default-accept-source-route
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-coredump-disable-backtraces
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-coredump-disable-storage
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-tipc-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-rp-filter
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv6-conf-all-accept-source-route
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-hfs-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-log-martians
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-all-secure-redirects
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-log-martians
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-grub2-slub-debug-argument
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-coredump-disable-backtraces
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-bluetooth-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-accept-source-route
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-all-secure-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-all-send-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-all-send-redirects
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-freevxfs-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-squashfs-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-vfat-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-default-secure-redirects
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-conf-default-secure-redirects
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-can-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-sysctl-net-ipv4-tcp-syncookies
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-kernel-module-hfsplus-disabled
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-grub2-audit-backlog-limit-argument
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: workers-scan-kernel-module-hfsplus-disabled
    scanName: workers-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-grub2-slub-debug-argument
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-grub2-audit-backlog-limit-argument
    scanName: masters-scan
    type: MachineConfig
  - apply: false
    remediationName: masters-scan-sysctl-net-ipv4-conf-all-accept-redirects
    scanName: masters-scan
    type: MachineConfig
  scanStatuses:
  - name: workers-scan
    phase: DONE
    result: NON-COMPLIANT
  - name: masters-scan
    phase: DONE
    result: NON-COMPLIANT

I can also download the XML results using the oc extract cm/masters-scan-master1.fqdn.pod

When I try and convert this to a html report, i get this issue:

$ oscap xccdf generate report results > results.html
No cdf:Benchmark ID specified and no suitable candidate has been autodetected.
OpenSCAP Error:: Could not apply XSLT /usr/share/openscap/xsl/xccdf-report.xsl to XML file: results [/builddir/build/BUILD/openscap-1.3.2/src/source/xslt.c:177]

Is there any additional steps I would need to do in order to convert the XML to a html report?

Also, the ARF file results that were generated by the scans seem to be garbage/corrupted in some way. When I try to extract the masters-scan-master1.fqdn-pod.xml.bzip2 file bzip2 throws an error that it is not a valid bzip file.

JAORMX commented 4 years ago

The file is bziped and base64ed. So you need to decode the CM's contents with base64 and then decompress it.

Alternatively, there is a PV that contains the bziped files. We're still working on a better way to expose the raw results so folks can consume that. But, for now, do oc get pv, you can mount that on a pod and then extract it.

jaedolph commented 4 years ago

Hi @JAORMX, thanks for the quick response. I was able to parse the ARF report successfully after doing a base64 -d masters-scan-master1.fqdn-pod.xml.bzip2 > results.xml.bzip2 before extracting. Would it be worth documenting that the ARF files are base64 encoded in the README?

Still looks like there are issues with the xxcdf report from the ConfigMap though

jhrozek commented 4 years ago

Hi @JAORMX, thanks for the quick response. I was able to parse the ARF report successfully after doing a base64 -d masters-scan-master1.fqdn-pod.xml.bzip2 > results.xml.bzip2 before extracting. Would it be worth documenting that the ARF files are base64 encoded in the README?

I opened PR #214 to address this.

Still looks like there are issues with the xxcdf report from the ConfigMap though

What issue exactly? The one with the oscap xccdf generate ?

jaedolph commented 4 years ago

What issue exactly? The one with the oscap xccdf generate

Yes, I cannot convert the xccdf xml into a html report. Don't have this issue with the ARF file

jhrozek commented 4 years ago

What issue exactly? The one with the oscap xccdf generate

Yes, I cannot convert the xccdf xml into a html report. Don't have this issue with the ARF file

So, I asked the OpenSCAP developers and it seems that this is sort of expected. I'll paste the conversation here, there's nothing private in it anyway:

We have filed issues from people trying to generate reports based on our XCCDF results with oscap xccdf generate report . Can anyone check what could be wrong?
$ oscap xccdf generate report results > results.html
No cdf:Benchmark ID specified and no suitable candidate has been autodetected.
OpenSCAP Error:: Could not apply XSLT /usr/share/openscap/xsl/xccdf-report.xsl to XML file: results [/builddir/build/BUILD/openscap-1.3.2/src/source/xslt.c:177]
the xccdf is produced from arf using ds-split

Jan Cerny,
I think it needs either ARF or XCCDF file that contains the original Benchmark

Jakub Hrozek,
Sounds plausible, does it mean that the XCCDF as produces by the ds-split is too bare-bone and to generate results, one should just use the ARF in the first place?

Jan Cerny,
Yes, I think so. To generate the HTML report oscap needs both the file with results and the input content with input rules. But, the file extracted by oscap ds-rds split contains only the results, doesn't contain input rules. To generate HTML report by oscap xccdf generate report  you need to pass either ARF or full XCCDF results (created by oscap xccdf eval --results). Using ARF has an advantage that the generated HTML will contain also the details about objects found on the scanned system.

Jakub Hrozek,
great, thanks a lot. I'll pass this along to the ticket
jhrozek commented 4 years ago

So, maybe we just need to document that in order to generate results, you need to use the ARF? btw there is a reason we split the XCCDF to be so small, it's because we need to correlate all the XCCDF results from all scans across machines that are scanned with a single ComplianceScan and we do it by putting the results in a ConfigMap (yeah..not the cleanest way, but we couldn't find anything better..). So it makes sense from that point of view to keep only the smallest possible subset of results in the XCCDF.

JAORMX commented 4 years ago

Is this still an issue?

hugohdz89 commented 4 years ago

Hello all, just to share what I did to get the HTML report.

$ oc exec pods/pv-extract -- ls /workers-scan-results/0 ocp4-cis-api-checks-pod.xml.bzip2 $ oc cp pv-extract:/workers-scan-results .

Then

$ bunzip2 -c ocp4-cis-api-checks-pod.xml.bzip2 > ocp4-cis-api-checks-pod.xml

$ yum install -y openscap openscap-scanner

$ oscap xccdf generate report ocp4-cis-api-checks-pod.xml > ocp4-cis-api-checks-pod.html

With above steps I could convert the XML into HTML report.

image

JAORMX commented 4 years ago

@hugohdz89 by the way, if it helps, here's a tool that'll help download the reports as well https://github.com/JAORMX/oc-compliance