mthalman
commented
3 months ago Ideally we would minimize changes to the actual pipeline to do this testing. The intent is that the validation pipeline behaves the same as the official pipeline. So I'd like to avoid injecting steps into the existing stages that are used by the validation and official pipeline. One possibility is to introduce a new stage at the end that is only used by the validation pipeline. It would include steps which does this kind of validation, either by examining artifacts or querying AzDO to get log output of the current build.
ellahathaway
MichaelSimons
Currently, we validate changes to the SB release pipeline by running PRs for changes against fake release runs. If the test pipeline run succeeds, then the changes are considered "successful". This type of validation falls into the category of functional testing where it's about validating that the software behaves as expected and works, but it doesn't check the actual content or internal structures.
While this type of testing is important, we should consider writing tests and validating changes against the actual content of the pipeline. For example, we should write tests that address the following questions: 1) Does the created PR contain the expected content 2) Does the pipeline correctly find an SDK diff run with the expected tag 3) Is the release info in "Read Release Info" what we expect 4) Are the associated pipeline runs the expected runs 5) Does the draft announcement contain the correct content 6) Is the mirrored tag in the expected repo 7) Does the mirrored tag point to the correct commit 8) Does the 6.0 PR get created, does it point to the expected commit 9) Do the download paths for the artifacts contain the artifacts that we expect