Show diff of different scan results; compare results of two scans

[gregelin]

Users can see what has changed between current scan, previous scan, and initial scan so they can see progress.

• What previously passing rules are now failing (most important)
• What rules previously failing are now passing
• Additions (new rules selected for the profile)
• Subtractions (rules previously selected and now unselected for the profile)

[gregelin]

Strategy for enabling this feature. This feature has the following elements:

• Create xslt template to compare two results.xml files using the document() function
  • create template
  • create xslt parameters and command
• Identify two most recent scans
• govready diff command
  • the command and functions
  • the xslt template being created during govready init
  • error handling
  • documentation

Strategy: Develop xslt template first and make sure diffing works. Then figure out the xsltproc command. Finally figure out how to pass around parameters and finding most recent versions of files.

[gregelin]

Information on finding most recent versions of files. Use a command a bash piped command combining ls prefix-result*.xml, sort, head -1, and sed.

[gregelin]

After more attempts of testing with document and variables, I am not really sure if an item by item comparison can be made purely by using xslt processing and document() command.

Going to try looping through in bash instead.

[gregelin]

Here is a simple loop using output of one xsltproc to power a second xsltproc.

for ruleid in `xsltproc listfail.xsl scans/results.xml`; do xsltproc --stringparam ruleid "${ruleid}" ruleinfo2.xsl scans/results.xml; done

[gregelin]

Bash compare:

compare.sh file:

#!/bin/bash

# usage:   bash ./compare.sh newresults.xml oldresults.xml
# examples
#   compare most recent result to an old scan:
#      bash ./compare.sh scans/results.xml scans/server-results-0910-1927.xml
#
#   compare new scan of failing to see what might have previously passed:
#      bash ./compare.sh scans/server-results-0910-1927.xml scans/results.xml

# For nice printing
_GREEN="\e[1;32m"
_RED="\e[1;31m"
_YELLOW="\e[1;33m"
_NORMAL="\e[0m"

newresults=$1
oldresults=$2

cmd="xsltproc listfail.xsl $oldresults"
rules=$(eval $cmd)

#echo "failed rules from $oldresults"
#echo $rules
printf "Comparing fails from two results files\n"
printf "file 'A' is $newresults\n"
printf "file 'B' is $oldresults\n\n"

for ruleid in $rules; do
    # get result for a ruleid from new results
    cmd_newresults='xsltproc --stringparam ruleid "${ruleid}" ruleidresult.xsl $newresults'
    newtestresult=$(eval $cmd_newresults)

    # get result for a ruleid from old results
    cmd_oldresults='xsltproc --stringparam ruleid "${ruleid}" ruleidresult.xsl $oldresults'
    oldtestresult=$(eval $cmd_oldresults)

    # print out comparison
    if [ $newtestresult = "pass" ]; then
        printf "A: ${_GREEN} $newtestresult${_NORMAL} "
    else
        printf "A: ${_RED} $newtestresult${_NORMAL} "
    fi

    if [ $oldtestresult = "pass" ]; then
        printf "A: ${_GREEN} $oldtestresult${_NORMAL} "
    else
        printf "B: ${_RED} $oldtestresult${_NORMAL} "
    fi

    printf "rule: $ruleid\n"

    #if [ $oldtestresult != $newtestresult ]; then
    #   printf "A: ${newtestresult} where B: ${oldtestresult} for rule: $ruleid"
    #fi
done

listfail.xsl file:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:cdf="http://checklists.nist.gov/xccdf/1.1"
    xmlns:exsl="http://exslt.org/common"
    xmlns:db="http://docbook.org/ns/docbook"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns="http://docbook.org/ns/docbook"
    xmlns:s="http://open-scap.org/"
    exclude-result-prefixes="xsl cdf db s exsl"
    xmlns:ovalres="http://oval.mitre.org/XMLSchema/oval-results-5"
    xmlns:sceres="http://open-scap.org/page/SCE_result_file"
    >

<!--
****************************************************************************************

Copyright: Greg Elin, 2014

This style sheet lists all failed rules

usage: $> xsltproc listfail.xsl result-file-name.xml

-->

<xsl:template match="/">
    <xsl:apply-templates/>
</xsl:template>
<xsl:output method="text" encoding="utf-8" />

<xsl:strip-space elements="*"/>
<xsl:output methood="text" encoding="UTF-8" />

<xsl:template match="/">

<xsl:for-each select="cdf:Benchmark/cdf:TestResult/cdf:rule-result[cdf:result = 'fail']">
<xsl:value-of select="@idref"/><xsl:text>
</xsl:text>
</xsl:for-each>

</xsl:template>

<!-- include to stop leakage from builtin tempaltes -->
<xsl:template match='node()' mode='engine-results'/>
<xsl:template match="text()"/>

</xsl:stylesheet>

ruleidresult.xsl content:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.1"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:cdf="http://checklists.nist.gov/xccdf/1.1"
    xmlns:exsl="http://exslt.org/common"
    xmlns:db="http://docbook.org/ns/docbook"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns="http://docbook.org/ns/docbook"
    xmlns:s="http://open-scap.org/"
    exclude-result-prefixes="xsl cdf db s exsl"
    xmlns:ovalres="http://oval.mitre.org/XMLSchema/oval-results-5"
    xmlns:sceres="http://open-scap.org/page/SCE_result_file"
    >

<!--
****************************************************************************************

Copyright: Greg Elin, 2014

This file provides a relitvely simple xslt transformation on openscap results.xml file
to show information regarding a single rule.

usage: $> xsltproc - -stringparam paramname paramvalue ruleidresult.xsl result-file-name.xml

-->

<xsl:param name="ruleid">notdefined</xsl:param>
<xsl:param name="compare_result">notdefined</xsl:param>
<xsl:strip-space elements="*"/>
<xsl:output method="text" encoding="utf-8" />

<xsl:template match="/">
    <xsl:apply-templates/>
</xsl:template>

<xsl:template match="cdf:rule-result">
<xsl:if test="@idref=$ruleid">
    <xsl:value-of select='cdf:result'/>
<xsl:text>
</xsl:text>
</xsl:if>
</xsl:template>

<!-- include to stop leakage from builtin tempaltes -->
<xsl:template match='node()' mode='engine-results'/>
<xsl:template match="text()"/>

</xsl:stylesheet>

[gregelin]

Add xsl templates templates/filterresults.xsl, templates/ruleidresults.xsl and script/compare.sh for making rule by rule comparisons between result xml files. See a8f018d0c126cc0335637469d233f159073c64c9

Next step is to integrate into govready commands.

[gregelin]

Diff method is now implemented in govready compare command. compare is a more accurate description because we are comparing two results files. commit: 928be637a6a41183b6a15018bc9b124fe46f9f34

Example:

# compare most recent results with earlier result across all selected controls
govready compare scans/results.html scans/test-results-0913-1754.xml

# compare most recent results with earlier result where most recent result passed
# compare current passes against previous result to see what improved
govready compare scans/results.html scans/test-results-0913-1754.xml fail

# compare most recent results with earlier result where most recent result failed
# compare current fails against previous result to see if anything failing previously passed
govready compare scans/results.html scans/test-results-0913-1754.xml fail

Starting final testing.

[gregelin]

Fixed final template bug. c7fd1df788c315cedcd0e938788c7e541d2273c0