search

GSA / fedramp-automation

FedRAMP Automation
https://www.fedramp.gov/using-the-fedramp-oscal-resources-and-templates/
Other
279 stars 85 forks source link

Inconsistent behavior for information type security level constraints from use of `|` or operator #772

Open aj-stein-gsa opened 3 days ago

aj-stein-gsa commented 3 days ago

This relates to ...

What happened?

We have a common pattern in NIST and FedRAMP's OSCAL constraints separately. For the FedRAMP constraints, the eager use of the | operator for grouping and testing of multiple roles is problematic. Today, we learned of inconsistency with how these behave, much like the legacy Schematron constraints did (because the underlying XPath 3 assumption is the same; I cannot find the exact relevant details in the W3C spec quickly).

https://github.com/GSA/fedramp-automation/blob/9d50039e52d2140fcce13940596474728dafe384/src/validations/constraints/fedramp-external-allowed-values.xml#L377-L389

This issue is to address that with a bug fix here, then integrate changes to the style guide in #760 or elsewhere if appropriate to strictly limit the use of this construct for similar use cases by developers. See the repro bug branch for details.

Commit on repro branch: https://github.com/aj-stein-gsa/fedramp-automation/commit/0248864c3b8a05676ca88888d76bedbd66a12658 Failure on CI/CD run of full test suite: https://github.com/aj-stein-gsa/fedramp-automation/actions/runs/11282608164/job/31380414397

Thanks to @DimitriZhurkin for locating and reporting this bug to the group today with some questions inadvertently during standup.

Relevant log output

me@computer fedramp-automation % git branch --show-current
772-pipe-operator-bug-repro
me@computer fedramp-automation % git rev-parse HEAD       
0248864c3b8a05676ca88888d76bedbd66a12658
me@computer fedramp-automation % make init && make test
git submodule update --init --recursive
Installing node modules...
npm install

up to date, audited 371 packages in 633ms

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npx oscal@latest use latest
Installing version: 2.2.0
Downloading OSCAL CLI...
Downloading version 2.2.0 from https://repo1.maven.org/maven2/dev/metaschema/oscal/oscal-cli-enhanced/2.2.0/oscal-cli-enhanced-2.2.0-oscal-cli.zip
Extracting OSCAL CLI...
Setting executable permissions for CLI at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
Creating OSCAL CLI symlink: oscal-cli => /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
OSCAL CLI installed to /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli
Alias created at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/bin/oscal-cli

up to date, audited 371 packages in 581ms

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npx oscal@latest use latest
Installing version: 2.2.0
Downloading OSCAL CLI...
Downloading version 2.2.0 from https://repo1.maven.org/maven2/dev/metaschema/oscal/oscal-cli-enhanced/2.2.0/oscal-cli-enhanced-2.2.0-oscal-cli.zip
Extracting OSCAL CLI...
Setting executable permissions for CLI at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
Creating OSCAL CLI symlink: oscal-cli => /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
OSCAL CLI installed to /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli
Alias created at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/bin/oscal-cli
src/web/module.mk:10: *** node.js version `v20.16.0` required, `v20.17.0` installed.  Stop.
me@computer fedramp-automation % npm run constraint security-level

> fedramp-automation@0.0.1 constraint
> node ./src/scripts/dev-constraint.js security-level

Found 64 constraints.
Selected 1 constraints for analysis.

Constraint Analysis and Test Execution:
security-level: Fully covered
//home/me/code/fedramp-automation/features/fedramp_extensions.feature {
  pass: 'security-level-PASS.yaml',
  fail: 'security-level-FAIL.yaml',
  fail_file: 'security-level-FAIL.yaml',
  pass_file: 'security-level-PASS.yaml'
} security-level
..Processing test case:Test Invalid Security Level Use
Description: Test that the specified security level is invalid.
Loaded content from: //home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml
oscal-cli validate //home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml --disable-schema-validation -c //home/me/code/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c //home/me/code/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-19dc296f-cd7d-4f49-8710-9b907bd5f2e4.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file://home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml' is invalid.

Checking status of constraint: security-level expecting: fail
Received: 13 matching mixed results (8 pass, 5 fail)
F

Failures:

1) Scenario: Validating OSCAL documents with metaschema constraints # features/fedramp_extensions.feature:142
   ✔ Given I have Metaschema extensions documents # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:86
       | filename                            |
       | fedramp-external-allowed-values.xml |
       | fedramp-external-constraints.xml    |
       | oscal-external-constraints.xml      |
   ✔ When I process the constraint unit test "security-level-FAIL.yaml" # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:94
   ✖ Then the constraint unit test should pass # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:103
       AssertionError: Negative test failed: Test failed with the following errors:
       security-level: invalid results received. 62% passing. Expected: pass_count "none", fail_count "all". Actual: 8 pass, 5 fail.
       : expected 'fail' to equal 'pass'
           + expected - actual

           -fail
           +pass

           at World.<anonymous> (file:////home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:110:34)
           at fulfilled (file:////home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:4:58)
           at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

1 scenario (1 failed)
3 steps (1 failed, 2 passed)
0m06.805s (executing steps: 0m06.795s)
Cucumber test for security-level failed: Command failed: NODE_OPTIONS="--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node" npx cucumber-js "//home/me/code/fedramp-automation/features/fedramp_extensions.feature:142"
security-level: Test failed
me@computer fedramp-automation %

How do we replicate this issue?

Where, exactly?

In constraints processing with constraint of ID security-level, but I have seen this pattern used frequently elsewhere.

Other relevant details

Need to sort out best practices rules on use of the rules with combinatoric use of group | sets with @david-waltermire.

aj-stein-gsa commented 2 days ago

OK I learned I was very wrong about this issue yesterday, and more to follow. This issue is not with the paths, the failure is occurring with schema enforcement. Takeaways from this:

So in the interim, we can refactor the test and improve the test harness it seems.