Add support for parallel test stages

[gvnabf]

Agent pool: Private Pool Agent specification: macOS Context: Android Test result format: jUnit

Describe the bug Parallel test stages with publish test results screenshot task on each stage attaches the screenshot multiple times. For example, the current # of stages that run in parallel is 3 and all of them have publish test screenshot task. Each also has publish test result task. Stage to finish first (let's call this test_stage0) will try to attach screenshot to its failed tests. Let's say 4 tests failed on this one. So it will try to attach 4 screenshots. This stage will succeed. Then, the stage to finish second (test_stage1) has the same failed tests as test_stage_0. The publish test result screenshot will detect 8 failed tests here including failed tests of test_stage0 since widget gets results by BuildId. Now it will attach the screenshots to the failed tests of both test_stage0 and test_stage1 since they have the same filename for the failed tests. test_stage0 now has 2 screenshots attach for each failed test. Same case will happen if stage to finish last (test_stage2) has the same failed tests as test_stage0 and test_stage1. Screenshot for test_stage2 will also be added to test_stage0 and test_stage1.

Total # of screenshots attached to each test failure on each test_stage: test_stage0 = 3 test_stage1 = 2 test_stage2 = 1

To Reproduce Steps to reproduce the behavior:

1. Setup a pipeline with 3 parallel stages. All should have publish test result and publish test screenshot and all should run the same test scripts but on different devices.
2. Run the pipeline.

Expected behavior Screenshot should only attach once to each failed step for every stage.

Noticed that test results are being fetched by BuildId. Perhaps we can also consider the test runId from PublishTestResults task since this is also part of the stage.

[eric-labelle]

Hi, I just had a quick look around and I can't seem to find if the currentRunId is available in their predefined variables. This seems feasible through their APIs to fetch all test results by runId but without this Id I would need to fetch all runs and iterate over them meaning our Task should be in a final stage after all parallel runs have completed. But then again, it would be back to move the current task that uses the buildId in a final stage.

This will definitively need a lot more thinking and experimentation in order to make sure to get the right screenshot from each run attached even if the same test fail in all 3 runs but the naming of the screenshot being independant from the pipeline this could be complex and would require some kind of altering of the screenshot names through the pipeline to include the runId.

Thanks for the suggestion though, I'll let you know if I think of some kind of workaround in the meantime.

[sphanley]

There's another version of this scenario, which is where instead of running all of your tests in parallel on multiple devices, you may be slicing your test suite to run multiple shards of it in parallel.

In this case, too, the task behaves in a suboptimal way if you don't collate the screenshots and upload them in a subsequent job: If more than one parallel stage has a failed test(s) and tries to do an upload, then they'll mark themselves as Succeeded w/ Issues, rather than Succeeded, when they try to upload the screenshots for test(s) which failed in a different shard.

For this kind of parallelization, there's a possible simpler solution: instead of assuming that a screenshot will be available for every test reported as failed for the build, would it be possible to instead simply have the task upload any screenshots which are present but gracefully ignore any which are missing, as they may be being handled by another shard? Maybe there could be a "Fail task on missing screenshots" type option to enable/disable this behavior for people who want it?

[sphanley]

A modified version of my suggested approach could also potentially help provide a solution for #2, as doing uploads based on which screenshots are present rather than based on the failure list could also help allow uploading all captured screenshots regardless of test result status.

[kjgen]

Closing issue because the extension is being deprecated and will no longer be maintained. An alternative solution is to use the Publish Test Result task which now supports file attachments.