cisagov / ScubaGear

Automation to assess the state of your M365 tenant against CISA's baselines
https://www.cisa.gov/resources-tools/services/secure-cloud-business-applications-scuba-project
Creative Commons Zero v1.0 Universal
1.66k stars 222 forks source link

Integrate automated test framework with release workflow #614

Open gdasher opened 1 year ago

gdasher commented 1 year ago

💡 Summary

Now that we have a solid automated test framework thanks to @crutchfield , we should integrate it with the release workflow. It seems like a candidate for a nested github action--we could define a github action that runs all the functional tests that could be triggered manually or by the workflow for building release candidates.

Then, the release workflow could be:

1) Build release candidate (signed) using workflow action, which would run all the functional tests and only build the candidate if they pass 2) Run manual tests 3) Mark release candidate as live release by removing the draft label.

Motivation and context

The idea is to avoid the situation where the build we test is not the same as the build we release and further standardize the release automation.

Implementation notes

Implementation should hopefully be some small github action YAML configuration given the release tests already support service principals and we use service principals in the smoke tests. We may need to add additional secrets with new credentials for other tenants and ensure the tenants are properly permissioned.

We would not be able to test non-service principal behavior in this way, though hopefully the difference there declines in the future as the sharepoint modules converge.

Acceptance criteria

We successfully do a release using this mechanism.

schrolla commented 2 months ago

May make sense to convert this to an epic and restate while flushing out more specific issues to define how to get there.

schrolla commented 2 months ago

Adding better release bundle testing would be good because the current "quick check" does not provide sufficient test coverage (e.g., does not exercise rego policies or MS APIs) to ensure operational capabilities of scuba.