[FEATURE REQUEST] Allow TestsNotExecuted to show as a non-fatal warning

[PeterConijn]

Is your feature request related to a problem? Please describe. In order to speed up our automated tests, we have split the tests into separate batches as per this blog. The author is awesome.

The problem is that you never quite know how many batches you get. I have enhanced the install code in the blog a little so that when I publish the test app, it automatically generates however many batches are needed. However, I cannot find a way to do this in the yml (this is sort of related to #725) so I need to hard-code it there. I am looking to keep this slightly better maintainable.

Describe the solution you'd like Since automatic detection is difficult, I was thinking to add one more testsuite batch than we currently have. However, this would currently fail the build, since "No Tests Were Executed" counts as a failure.

Is it possible to somehow create a setting to show this as informational and not have it list the build as failed or partially successful, but as successful? That was I can monitor and as soon as the message no longer appears (which means our tests now have expanded to this batch, I can update the pipeline with the next (empty) batch.

It's not ideal, but better than having tests that are not run in the pipeline because I was too late in appending the test batches.

Describe alternatives you've considered Right now, we manually publish the tests locally and monitor the batches, but with the number of devs and human nature, the risk of forgetting it is high.

Additional context Test suite batches

Yml snippet to run the tests

[waldo1001]

We'll add a setting.

[waldo1001]

We added a setting "allow_zero_tests".

[waldo1001]

Can you try:

- task: Hodor.hodor-alops.alopsapptest.ALOpsAppTest@1
  displayName: 'ALOps App Test'
  inputs:
    allow_zero_tests: true