Evangelink
commented
8 months ago What do you envision the purpose of that hierarchy is, and how would it be reported to the user in Test Explorer? xUnit.net, for example, can make a case for its hierarchy being Assembly > Test Collection > Test Class > Test Method > Test Case > Test. Would it make sense for us to report all of that? Some of that? None of that?
We have designed this new testing principles by trying to be as open as possible to the various scenario we could see in testing in general (not .NET specific) and it seems that supporting a tree of tests is what's most generic as it allows flat list (as VSTest), test group and tests (as in F# Expecto for example), it supports BDD (like SpecFlow) and also Behavior Trees like automation/Unity... And so we have done something similar for the protocol.
The main problem is that Test Explorer (VS and VS Code) doesn't support this. We have some discussions with the team owning this part and we hope we will be able to make this component evolve in the future but there is currently no short-term plan for it.
So as of now, we have the support for this hierarchical feature in the platform and the protocol but no way to display it. We are considering alternatives if there is no way to make Test Explorer move quickly but this is still under discussion.
Would I report the assembly as "in progress", then at the end, report it as "failed" or "passed" based on whether any tests failed?
If we remove any existing constraint, we are still having some discussions about that. My thinking is that I would want any test framework to be able to have full control/customization so yes you would/could report any node as anything based upon the logic you want. Some concerns were raised that devs are used to the current model and that they would expect similar behavior across test frameworks.
The API description of test node includes a node type of action vs. group (it would make sense that everything that's not a running test would be a group), but I don't see that reflected into the TestNode object in Microsoft.Testing.Platform. I'm presuming that every TestNode is implicitly an action node type right now; is that correct? So is it not possible to reporting groupings today?
Because of the current limits of Test Explorer and because the main usage for now are matching old models, we are short-circuiting and considering any node to be an action. But the idea we had is to say this would be another property of the reported nodes.
When reporting
TestNodeupdates, you can include the parent test node UID. Looking at the code inMicrosoft.Testing.Extensions.VSTestBridgefor examples, it doesn't appear that any reporting there is using any hierarchy.What do you envision the purpose of that hierarchy is, and how would it be reported to the user in Test Explorer? xUnit.net, for example, can make a case for its hierarchy being Assembly > Test Collection > Test Class > Test Method > Test Case > Test. Would it make sense for us to report all of that? Some of that? None of that?
Experimentation with
dotnet testtoday shows that none of that appears to be used at the moment. Reporting parentless test results, just like the VSTestBridge, results in successful execution:I have not attempted to add any hierarchy (this is just experimental work at the moment), but I'm not sure how
dotnet testwould interpret that currently. Let's assume a single level of hierarchy (Assembly > Test) was supported. Would I report the assembly as "in progress", then at the end, report it as "failed" or "passed" based on whether any tests failed? How would that contribute to the test count indotnet test? The API description of test node includes a node type ofactionvs.group(it would make sense that everything that's not a running test would be a group), but I don't see that reflected into theTestNodeobject inMicrosoft.Testing.Platform. I'm presuming that everyTestNodeis implicitly anactionnode type right now; is that correct? So is it not possible to reporting groupings today?