GSA / fedramp-automation

FedRAMP Automation
https://www.fedramp.gov/using-the-fedramp-oscal-resources-and-templates/
Other
277 stars 85 forks source link

Validating SAP, SAR, and POAM using the fedramp-automation validation code #407

Open Telos-sa opened 1 year ago

Telos-sa commented 1 year ago

Describe the bug

When attempting to validate the SAP and the SAR using the fedRAMP validator, the tool is processing using SSP references. There is no way to direct the tool to instead reference the SAP and SAR validation. This is causing failure to process.

Who is the bug affecting?

Anyone who wants to validate their data using the FedRAMP validator tool.

Is this report specifically related to the Word or Excel files from fedramp.gov?

using both XML and JSON versions of the SAP and SAR

If so, please do not open an issue here. Follow the guidance in this repository's README and contact info@fedramp..gov.

What version of OSCAL are you using? OSCAL 1.0.0 > 1.0.4

What is affected by this bug?

the fedramp-automation validation tool.

When does this occur?

Anytime validation is attempted against models other than SSP (SAP, SAR, POAM) With both XML and JSON files. NOTE: Tested Json even though the validator requests to not use json.

How do we replicate the issue?

  1. git the current fedramp-automation repository.
  2. follow steps to make the validator
  3. run: ./bin/validate_with_schematron.sh -f ~/fedramp-automation/filesForValidation/coalFireSAP.xml {If applicable, add screenshots to help explain your problem.}

Errors: output dir report/schematron doc requested to be validated: /home/ec2-user/fedramp-automation/filesForValidation/coalFireSAP.xml using saxon version 10.8 Saxon JAR at classpath ./../../vendor/Saxon-HE-10.8.jar is valid preprocessing stage 1: rules/ssp.sch to: ./target/ssp-stage1.sch preprocessing stage 2: ./target/ssp-stage1.sch to: ./target/ssp-stage2.sch compiling: rules/ssp.sch to: ./target/ssp.xsl validating doc: /home/ec2-user/fedramp-automation/filesForValidation/coalFireSAP.xml with rules/ssp.sch output found in report/schematron//home/ec2-user/fedramp-automation/filesForValidation/coalFireSAP.xml__ssp.results.xml Warning at char 21 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  The required item type of the first operand of '|' is node(), but the supplied   expression {docOrder(docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and   @value eq "web-application"]]))/@uuid)!xs:string(.)} has item type xs:string. The   expression can succeed only if the supplied value is an empty sequence. Warning at char 149 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  The required item type of the second operand of '|' is node(), but the supplied   expression {docOrder(docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and   @value eq "web-application"]]))/@uuid)!xs:string(.)} has item type xs:string. The   expression can succeed only if the supplied value is an empty sequence. Warning at char 17 in xsl:variable/@select on line 984 column 270 of ssp.xsl:   SXWN9000  The required item type of the first operand of '|' is node(), but the supplied   expression {docOrder(docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and   @value eq "role-based"]]))/@uuid)!xs:string(.)} has item type xs:string. The expression   can succeed only if the supplied value is an empty sequence. Warning at char 136 in xsl:variable/@select on line 984 column 270 of ssp.xsl:   SXWN9000  The required item type of the second operand of '|' is node(), but the supplied   expression {docOrder(docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and   @value eq "role-based"]]))/@uuid)!xs:string(.)} has item type xs:string. The expression   can succeed only if the supplied value is an empty sequence. Warning at char 21 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  The only value that can pass type-checking is an empty sequence. The required   item type of the first operand of '|' is node(), but the supplied expression   {docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.)} has item type xs:string Warning at char 149 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  The only value that can pass type-checking is an empty sequence. The required   item type of the second operand of '|' is node(), but the supplied expression   {docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.)} has item type xs:string Warning at char 21 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  Required item type of value of variable $ssp-web-apps is xs:string; supplied   value (docOrder(docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.)) |   docOrder(docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.))) has item type node(). The expression can   succeed only if the supplied value is an empty sequence. Warning at char 17 in xsl:variable/@select on line 984 column 270 of ssp.xsl:   SXWN9000  The only value that can pass type-checking is an empty sequence. The required   item type of the first operand of '|' is node(), but the supplied expression   {docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and @value eq   "role-based"]]))/@uuid!xs:string(.)} has item type xs:string Warning at char 136 in xsl:variable/@select on line 984 column 270 of ssp.xsl:   SXWN9000  The only value that can pass type-checking is an empty sequence. The required   item type of the second operand of '|' is node(), but the supplied expression   {docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and @value eq   "role-based"]]))/@uuid!xs:string(.)} has item type xs:string Warning at char 21 in xsl:variable/@select on line 574 column 291 of ssp.xsl:   SXWN9000  The only value that can pass type-checking is an empty sequence. The required   item type of the value of variable $ssp-web-apps is xs:string, but the supplied expression   {docOrder(docOrder($ssp-doc/(component[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.)) |   docOrder(docOrder($ssp-doc/(inventory-item[prop[xs:string(@name) eq "type" and @value eq   "web-application"]]))/@uuid!xs:string(.))} has item type node()

Expected behavior (i.e. solution)

The validator is model agnostic. either need to pass in what model to run against during the initialization command, or have the validator review the file to determine which model to validate.

Must be able to generate results for all models, from a validator that can be installed locally, to prevent leaking of confidential data.

Telos-sa commented 1 year ago

@Telos-sa, the validator needs to be reviewed, needs work around since it is not an "out of the box". For the validators. Will test again with clean install.

Here are the installs that we have done on aws linux2.

Install depend (may be others, since I have different baseline). Would be great if we could toss this in a container. Though I have not tested that yet.

sudo yum install git -y

sudo yum install git-all

git clone https://github.com/GSA/fedramp-automation.git

cd fedramp-automation/vendor/

git clone https://github.com/xspec/xspec.git

git clone https://github.com/Schematron/schematron.git

sudo yum install gcc-c++

sudo yum update

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

. ~/.nvm/nvm.sh

nvm install v18.15.0

nvm install 16

Build

cd fedramp-automation

make init-repo

cd src/web/

npm install

npm run build

Restructure based on commands in code

mv ~/fedramp-automation/src/validations/rules/rev4/ssp.sch ~/fedramp-automation/src/validations/rules/ssp.sch

mv -r /fedramp-automation/dist/content/ /fedramp-automation/content/

cp -r ~/fedramp-automation/dist/content/ /fedramp-automation/content/

cp -r ~/fedramp-automation/dist/content/ ~/fedramp-automation/content/

cp -r ~/fedramp-automation/dist/ ~/dist/

Run validator against example to confirm functionality.

cd fedramp-automation/src/validations/

./bin/validate_with_schematron.sh -f ~/fedramp-automation/src/content/rev4/templates/ssp/xml/FedRAMP-SSP-OSCAL-Template.xml

Change directory and check reporting

cd report/schematron/home/ec2-user/fedramp-automation/src/content/rev4/templates/ssp/xml

Telos-sa commented 1 year ago

instead of make init-repo. Will tests doing make clean, then make all.