Open gdasher opened 1 year ago
May make sense to convert this to an epic and restate while flushing out more specific issues to define how to get there.
Adding better release bundle testing would be good because the current "quick check" does not provide sufficient test coverage (e.g., does not exercise rego policies or MS APIs) to ensure operational capabilities of scuba.
💡 Summary
Now that we have a solid automated test framework thanks to @crutchfield , we should integrate it with the release workflow. It seems like a candidate for a nested github action--we could define a github action that runs all the functional tests that could be triggered manually or by the workflow for building release candidates.
Then, the release workflow could be:
1) Build release candidate (signed) using workflow action, which would run all the functional tests and only build the candidate if they pass 2) Run manual tests 3) Mark release candidate as live release by removing the draft label.
Motivation and context
The idea is to avoid the situation where the build we test is not the same as the build we release and further standardize the release automation.
Implementation notes
Implementation should hopefully be some small github action YAML configuration given the release tests already support service principals and we use service principals in the smoke tests. We may need to add additional secrets with new credentials for other tenants and ensure the tenants are properly permissioned.
We would not be able to test non-service principal behavior in this way, though hopefully the difference there declines in the future as the sharepoint modules converge.
Acceptance criteria
We successfully do a release using this mechanism.