Constraint Script Runs More Than One Constraint

Gabeblis commented 1 day ago

This relates to ...

[X] the FedRAMP OSCAL Validations

What happened?

When running the constraint script, (npm run constraint <constraint-id>), it runs more than one constraint. I noticed that this behavior happens when another constraint id ends with the constraint id that you are running.

Relevant log output

$ npm run constraint cloud-service-model

> fedramp-automation@0.0.1 constraint
> node ./src/scripts/dev-constraint.js cloud-service-model

Found 74 constraints.
Selected 1 constraints for analysis.

Constraint Analysis and Test Execution:
cloud-service-model: Fully covered
..Processing test case:Test for cloud-service-model
Description: This test case validates the behavior of constraint cloud-service-model
Loaded content from: /workspaces/fedramp-automation/src/validations/constraints/content/ssp-cloud-service-model-INVALID.xml
oscal-cli validate /workspaces/fedramp-automation/src/validations/constraints/content/ssp-cloud-service-model-INVALID.xml --disable-schema-validation -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-8edda4ea-9abc-4cc5-ab46-4a97edcc5523.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file:/workspaces/fedramp-automation/src/validations/constraints/content/ssp-cloud-service-model-INVALID.xml' is invalid.

Checking status of constraint: cloud-service-model expecting: fail
Received: 1 matching fail results (0 pass, 1 fail)
.

1 scenario (1 passed)
3 steps (3 passed)
0m03.382s (executing steps: 0m03.368s)
..Processing test case:Test for cloud-service-model
Description: This test case validates the behavior of constraint cloud-service-model
Loaded content from: /workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml
oscal-cli validate /workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-da7488f6-abd0-493a-9a9f-fff91ed0ad4a.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file:/workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml' is invalid.

Checking status of constraint: cloud-service-model expecting: pass
Received: 1 matching pass results (1 pass, 0 fail)
.

1 scenario (1 passed)
3 steps (3 passed)
0m08.832s (executing steps: 0m08.819s)
..Processing test case:Negative Test for has-cloud-service-model
Description: Test that an SSP system-characteristics element does not have a prop element with a name attribute equal to "cloud-service-model".
Loaded content from: /workspaces/fedramp-automation/src/validations/constraints/content/ssp-has-cloud-service-model-INVALID.xml
oscal-cli validate /workspaces/fedramp-automation/src/validations/constraints/content/ssp-has-cloud-service-model-INVALID.xml --disable-schema-validation -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-550e70f5-6abc-4c18-af93-64133464979d.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file:/workspaces/fedramp-automation/src/validations/constraints/content/ssp-has-cloud-service-model-INVALID.xml' is invalid.

Checking status of constraint: has-cloud-service-model expecting: fail
Received: 1 matching fail results (0 pass, 1 fail)
.

1 scenario (1 passed)
3 steps (3 passed)
0m03.319s (executing steps: 0m03.300s)
..Processing test case:Positive Test for has-cloud-service-model
Description: Test that an SSP system-characteristics element has a prop element with a name attribute equal to "cloud-service-model".
Loaded content from: /workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml
oscal-cli validate /workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c /workspaces/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-7fe1351a-7666-45ca-98fa-84815381a6cc.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file:/workspaces/fedramp-automation/src/validations/constraints/content/ssp-all-VALID.xml' is invalid.

Checking status of constraint: has-cloud-service-model expecting: pass
Received: 1 matching pass results (1 pass, 0 fail)
.

1 scenario (1 passed)
3 steps (3 passed)
0m08.886s (executing steps: 0m08.865s)
Cucumber tests for cloud-service-model passed successfully.
cloud-service-model: Test passed

How do we replicate this issue?

Run npm run constraint cloud-service-model and then it will run that constraint and also has-cloud-service-model.

Where, exactly?

This occurs in the constraint script. Specifically, the cause is in dev-constraint.js under scripts in the develop branch.

Other relevant details

No response

aj-stein-gsa commented 1 day ago

@Gabeblis you know the fix and got something in mind? I just tested this behavior and it works the same for me. Moving to from Backlog to Ready.

Gabeblis commented 1 day ago

@Gabeblis you know the fix and got something in mind? I just tested this behavior and it works the same for me. Moving to from Backlog to Ready.

Yes, I have a fix in mind. We can use a bit of regex to clean up each line for an exact comparison, rather than using the includes method. This way, we ensure that only the exact constraint is matched and run, without including similar strings. It's a tiny change. I can open a PR for this when there is room.

GSA / fedramp-automation