Introduce a capability to report number of tests

[Evangelink]

Summary

Let the framework announce if it's capable of reporting the number of tests to run.

Background and Motivation

With the new live output, we currently display the number of passed, failed and skipped tests. Some framework could know upfront the number of tests to be executed and this information could be used to drive the number of tests.

Proposed Feature

Introduce a framework capability to report the number of tests that will be executed. This can be used to drive multiple UI features (progress bar, remaining tests count...)

Alternative Designs

None.

[thomhurst]

How would this work with filters? This'd be evaluated before filtering has been done wouldn't it?

[Evangelink]

The idea is to say that the capability is some optional "capacity" of a test framework, in this case to say that it knows how many tests need to be executed. The responsibility of knowing if that's possible or not lies on the test framework.

This being said you are right, we are seeing 2 main problems:

• filtering
• expansion of parameterized tests

for these 2 cases it's possible we don't know upfront how many tests in total there will be.

One potential idea raised by @MarcoRossignoli is to not report upfront the total number but to be able to report a variable remaining tests count. This way, as you progress/handle filtering or expansion you can increase or decrease what's left. Obviously, there is an impact on the kind of display we can show.

At the moment, this ticket serves more as a root of exploration because there might be no easy solution and/or the cost for value might not be worth it.

[thomhurst]

Instead of a capability could it not instead be a message pushed to the message bus? The framework has control over when to send it then, which could be after filtering?

[Evangelink]

This needs to be a message but we need a capability to know that the currently plugged framework supports this capability.

When initializing the console UI for the run, we need to know if the framework will be sending the count message or not to change how we display things. With the capability, we can say ok the framework will send the message so I can prepare my UI and adjust/adapt when I receive the message.

[MarcoRossignoli]

Instead of a capability could it not instead be a message pushed to the message bus? The framework has control over when to send it then, which could be after filtering?

This was also a discussed idea and I think it's the good one. Something simple like TestSessionProgress(int percentace, int testsRan, int? totalTests) adapters know better than us how to do it.