Test Harness and Framework for OSCAL CLI testing

Rene2mt commented 3 months ago

This is a ...

research - something needs to be investigated

This relates to ...

[ ] the FedRAMP OSCAL Registry
[ ] the FedRAMP OSCAL baselines
[ ] the Guide to OSCAL-based FedRAMP Content
[ ] the Guide to OSCAL-based FedRAMP System Security Plans (SSP)
[ ] the Guide to OSCAL-based FedRAMP Security Assessment Plans (SAP)
[ ] the Guide to OSCAL-based FedRAMP Security Assessment Results (SAR)
[ ] the Guide to OSCAL-based FedRAMP Plan of Action and Milestones (POA&M)
[ ] the FedRAMP SSP OSCAL Template (JSON or XML Format)
[ ] the FedRAMP SAP OSCAL Template (JSON or XML Format)
[ ] the FedRAMP SAR OSCAL Template (JSON or XML Format)
[ ] the FedRAMP POA&M OSCAL Template (JSON or XML Format)

User Story

As a developer, I need an automated testing approach to QA the OSCAL CLI whenever changes are made or new versions are released.

Goals

Problem Statement

The FedRAMP OSCAL automation team needs a good, light-weight, simple test harness and framework for automated unit testing. The solution will be used for QA of developed FedRAMP OSCAL validation rules integrated into the OSCAL CLI tool. The solution may also be used to integrate with testing of other FedRAMP OSCAL tools in the future.

Requirements

Licensing - Any leveraged automated (unit) testing frameworks must be open source.
Test Harness - Must support declarative approach to defining tests (e.g., new tests can be added with no/minimal coding).
- Should support use of unit test templates to make it easier to create new tests.
Design - Must support creating and categorizing comprehensive test scenarios covering different validation rules, rule combinations, and edge cases.
Configuration - Approach should simplify configuration of test vectors
Language / Platform - Preferably should be based on Node.js and Javascript/TypeScript, although other languages may be considered if there is a considerable benefit
Data - Must include both valid and invalid data samples to verify rule enforcement and error detection.
Execution / Results - must be able to (automate) running of tests defined in test harness and produce test results, including pass/fail status, error messages, and performance metrics.
- Must be able to run entire test suite or subset
Integration - Should integrate easily with CI/CD pipelines

In order to identify a solution, the dev team will:

[x] Identify various open-source options
[x] Document pros & cons of identified options
[x] Select one or two options for further investigation
[x] Proof-of-concept - implement a few, basic unit tests using the selected option
[x] Review / demo with dev team (determine if approach will work, what changes are needed, etc.)
[ ] Decide on options (continue or pivot to another alternative)
[ ] Document decision in ADR (see https://github.com/GSA/fedramp-automation/pull/593)

Dependencies

No response

Acceptance Criteria

All FedRAMP Documents Related to OSCAL Adoption (https://github.com/GSA/fedramp-automation) affected by the changes in this issue have been updated.
A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.

Other information

No response

Rene2mt commented 3 months ago

Team did a preliminary review of various testing frameworks including:

Mocha
Jest
Jasmine
Cucumber
AVA

After weighing the pros and cons of each, team decided to do a proof of concept with Jest.

Rene2mt commented 3 months ago

Need to:

[ ] Resolve test runner issue with asynchronous operations that weren't stopped in test...causing time out issues
[ ] Come up with data structure for the declarative test vector definition yaml file. The following is a preliminary example of the data elements we might need but needs to be refined:

tests: 
- test-content: oscal-ssp-file.json
  tag:
  - metadata
  - system-characteristics
  - system-implementation
  - control-implementation
  - back-matter
  expectation:
  - id: some-validation-id
    description: "some test description"
    command: "validate" # optional or not necessary at all if only testing oscal-cli validation
    level: ERROR
    location: /some/metapath/node
    result: pass
  - id: some-validation-id
    description: "some test description"
    command: "validate" # optional or not necessary at all if only testing oscal-cli validation
    level: ERROR
    location: /some/metapath/node2
    result: fail
  - id: some-other-validation-id
    description: "some test description"
    command: "validate" # optional or not necessary at all if only testing oscal-cli validation
    level: WARNING
    location: /some/metapath/node
    result: pass

Once a format is established, will need to update the test runner accordingly.

Rene2mt commented 3 months ago

OSCAL-CLI will be generating SARIF output - https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html

Rene2mt commented 2 months ago

YAML unit tests will follow this structure. Need naming convention and location for these declarative unit tests.

tests: 
- test-content: oscal-file.json
  tag:
  - metadata
  - system-implementation
  expectation:
  - id: some-validation-id
    level: ERROR
    location: /some/metapath/node
    result: pass
  - id: some-validation-id
    level: ERROR
    location: /some/metapath/node2
    result: fail
  - id: some-other-validation-id
    level: ERROR
    location: /some/metapath/node
    result: pass

Rene2mt commented 4 weeks ago

For full implementation details, see PR #622. In summary:

Testing harness is based on Cucumber, an open-source software testing tool that helps developers test software by running scenarios to see if they produce the intended results
Metaschema-based constraints for FedRAMP validation rules (in the /src/validations/constraints folder)

Declarative YAML unit tests to ensure developed constraints yield expected results (in the /src/validations/constraints/unit-tests folder). The YAML unit tests must use the following structure:

test-case:
name: {name / title for the constraint test scenario}
description: {a description of the constraint being tested}
content: {path to the OSCAL content the constraint will be tested on}
pipeline: {OPTIONAL element if some pipeline action such as profile resolution needs to happen on the specified content before executing the test}
- action: {OPTION action such as resolve-profile} 
expectations: 
- constraint-id: {the ID of the constraint being tested}
result: {pass | fail}

OSCAL content for the unit tests (in the /src/validations/constraints/contents folder)
A Node.js script is the test driver, taking the cucumber scenarios & steps, metaschema constraints, their corresponding unit tests, and specified unit test PASS and FAIL content files, and using that to execute OSCAL-CLI validations.

You can run make test at the root of your local fedramp-automation repository to setup the testing harness locally. More detailed instructions are forthcoming (see PR #638).

GSA / fedramp-automation