Test Explorer - Give frameworks full control over test explorer layout

[thomhurst]

I assume that the test explorer view in IDEs is driven by the TestMethodIdentifierProperty on TestNodes?

This provides no way to show parameter types or argument types.

In data driven testing, I'd say it's not uncommon to inject in different objects that inherit from a base, and then want to run only those particular tests. This doesn't seem to be possible currently.

For example, consider a multi-tenanted application, and you want to run tests against a particular tenant you'd made changes for.

In pseudo code, the test is set up like this:

class TenantBase { ... } class Tenant1 : TenantBase { ... } class Tenant2 : TenantBase { ... } class Tenant3 : TenantBase { ... }

[Inject<Tenant1>]
[Inject<Tenant2>]
[Inject<Tenant3>]
class MyTestClass(TenantBase tenant)

Meaning it is repeated for each tenant, with a different object being passed in.

Because the test explorer / test node properties have no visibility of the parameter types, or more importantly, the argument types, there is no way for it to display this in a nice way.

For example, in VSTest in NUnit, I could set up a test like this, and I'd get in the test explorer:

> MyTestClass
    > MyTestClass(MyNamespace.Tenant1)
    > MyTestClass(MyNamespace.Tenant2)
    > MyTestClass(MyNamespace.Tenant3)

This makes it super easy to selectively run the tests that I want.

---

I have tried this in my own framework. Code

new TestMethodIdentifierProperty
(
...
TypeName: testDetails.ClassType.Name,
...
)

[ClassDataSource(typeof(Derived1))]
[ClassDataSource(typeof(Derived2))]
public class ClassDataSourceDrivenTests2
{
    private readonly Base _base;

    public ClassDataSourceDrivenTests2(Base @base)
    {
        _base = @base;
    }

    [Test]
    public void Base_Derived1()
    {
        // Dummy method
    }

    [Test]
    public void Base_Derived2()
    {
        // Dummy method
    }

    public class Base {}
    public class Derived1 : Base {}
    public class Derived2 : Base {}
}

Text explorer view:

So you can see that it makes it very difficult - I don't actually know which one I'm running.

Also, this would technically affect the test methods too - While they have the parameter types, there's no way to pass in the argument types, so we wouldn't be able to differentiate. Of course, we can mitigate this with a custom display name, but this isn't something we can do for the class itself.

[thomhurst]

Here's a current VSTest NUnit repro:

[TestFixtureSource(typeof(Base), nameof(Base.All))]
public class ClassDataSourceDrivenTests2
{
    private readonly Base _base;

    public ClassDataSourceDrivenTests2(Base @base)
    {
        _base = @base;
    }

    [Test]
    public void Base_Derived1()
    {
        Console.WriteLine(_base.GetType());
    }

    [Test]
    public void Base_Derived2()
    {
        Console.WriteLine(_base.GetType());
    }

    public class Base 
    {
        public static IEnumerable<Base> All()
        {
            yield return new Derived1();
            yield return new Derived2();
        }
    }

    public class Derived1 : Base {}
    public class Derived2 : Base {}
}

Displayed Test Explorer: Rider:

Visual Studio:

[thomhurst]

I've tried setting the TypeName in TestMethodIdentifierProperty to more of a display name. That sort of works, but doesn't give you the same level of organisation of being able to group by class WITHOUT params and then WITH.

E.g.:

> MyClass
    > MyClass(Tenant1)
        - Test1
        - Test2
        - etc...
    > MyClass(Tenant2)
        - Test1
        - Test2
        - etc...
    > MyClass(Tenant3)
        - Test1
        - Test2
        - etc...

[thomhurst]

Thinking about this further, it'd be good to allow the framework to have full control over how tests display in test explorers. I think the best way to allow this would be a property that if provided, takes precedence over anything else (and simply falls back to existing behaviour if not provided)

That property can simply take a collection of paths. If any except the final path (because that evaluates to a single runnable test so then can't be expanded) are duplicated, then they will combine together in the test explorer.

For example:

public class TestExplorerDisplayProperty : IProperty
{
    public TestExplorerDisplayProperty(params string[] paths)
    {
        // ...
    }
}

And then in my data driven test, I could do:

new TestExplorerDisplayProperty
(
  [
    Test.Namespace,
    Test.BasicClassName,
    Test.ClassNameWithArgument,
    Test.BasicMethodName,
    Test.MethodNameWithArgument
  ]
)

Which would cause

> MyNamespace
  > MyClass
    > MyClass(Type1)
      > Test1
          Test1(Value1)
          Test2(Value2)
    > MyClass(Type2)
      > Test1
          Test1(Value1)
          Test2(Value2)

And for non-data driven tests I can just pass in fewer, more basic paths:

new TestExplorerDisplayProperty
(
  [
    Test.Namespace,
    Test.BasicClassName,
    Test.BasicMethodName,
  ]
)

Which would cause

> MyNamespace
  > MyClass
      Test1

[thomhurst]

In the existing NUnit example above, there's also differences between how Rider and Visual Studio display these. I think it's better to give this control to the testing framework rather than the IDE itself. (I know the IDE can obviously implement this behaviour however they like, but if there's a mechanism for the framework to control the layout it'd make sense for them to use it I think.) So this would fix differences/discrepencies between using different IDEs.

[Evangelink]

Thanks @thomhurst! This is something @MarcoRossignoli and I are trying to push since the inception of the new platform. Sadly Test Explorer is owned by a different team with different vision and priorities.

[thomhurst]

Feel free to send them this ticket if it'll help persuade them!

Currently the test explorer makes assumptions about how to organise tests. Different frameworks may have different intentions so giving that control can give a better experience for the user.

Heck, you could then even possibly allow users to group by test category or something in the test explorer. It could make things really interesting!

[Evangelink]

Currently the test explorer makes assumptions about how to organise tests. Different frameworks may have different intentions so giving that control can give a better experience for the user.

Entirely! Current Test Explorer enforces the shape and structure of everything when it should offer a common set of features but give flexibility to frameworks. The 4 levels is a huge limitation to me and I would like to be able to express my tests as it's done in F# Expecto where each group level is a "simple string" which leaves the user possibility to group tests functionally. This would also simplify things like BDD (e.g. SpecFlow).

[thomhurst]

Thanks for replying @Evangelink ! Completely agree with you. Hoping @MarcoRossignoli can throw some weight here too and maybe make this a possibility.

At work we have a multi-tenanted solution, and being able to run tests specifically for an injected class type (the tenant) would be a need for us. Much so that if went couldn't do it in the new world we'd just stick to existing VSTest NUnit. Hopefully that means something that there is a want for it and if we want to improve testing and give a framework that supports everything we need, the test explorer is part of that!

[thomhurst]

Also, you guys mention that the testing explorer is owned by another team, which I completely appreciate, but visual studio isn't the only testing explorer out there as you know, so is it a requirement to get approval from them first before implementing a generic mechanism that other tools could at least use?

[Evangelink]

Definitely not a requirement but for us to be approved to work on something that doesn't benefit to our products is another story :)

[thomhurst]

Don't suppose there was any discussions with the test explorer team since the last comment? cc. @MarcoRossignoli

[Evangelink]

There was some discussion but no move/change. We will be quite busy until the end of the month but I will bring back the discussion to the table after that.

[tom-englert]

We have the same issue, and the UX in test explorer should definitely be improved.

[rigtigeEmil]

Is there any update on the this issue?

@Evangelink do you happen to have insights into what the vision or priorities are for the test explorer? 😊

Thanks @thomhurst! This is something @MarcoRossignoli and I are trying to push since the inception of the new platform. Sadly Test Explorer is owned by a different team with different vision and priorities.

[Evangelink]

Pinging @webreidi @pavelhorak for awareness

[thomhurst]

Just to provide more scenarios/context:

Namespace > Class > Inner Class > Inner Inner Class > Test

Maybe not the most conventional but a clear example of a use case that can be very real, but also unsupported by the current limitations of the text explorer.

I honestly think it would be super simple to introduce a property called something like TestExplorerSectionsProperty and that can include every expandable group, and then the final node in the tree will be the TestNode display name.

Example:

new TestNode
{
    DisplayName = "MyTest",
    Properties = 
    [
        new TestExplorerSectionsProperty
        (
        [
            "Namespace",
            "Class",
            "Inner Class",
            "Inner Inner Class",
        ]
        ),
    ]
}

I know it would need to be implemented by all the relevant IDEs, but the hardest part is surely implementing the UI and structure etc? Determining the hierarchy just sounds like dictionaries that can also contain dictionaries.

For example, if I had:

public class MyTests : TestBase
{
    [Test]
    public void Test()
    {
        ...
    }

    public class InnerClass2
    {
        [Test]
        public void Test()
        {
            ....
        }
    }

        public class InnerClass3
        {
            [Test]
            public void Test()
            {
                ....
            }
        }
}

This could make sense to a user to be able to organise sub-tests, or tests relating to a single feature, but with the limitations of the test explorer they wouldn't all be listed together.

Of course this still relies on the test framework to set the relevant properties, but at least I could look at the full class name and split them on the + character. And then imagine they were grouped by MyTests then InnerClass and then InnerClass2. That makes things so much cleaner/simpler for the user!

This is also just one example of how to structure them.

I could also expose an attribute to allow users to control their own structure. I think that allows a greater and more personalised testing experience based on user needs.

Currently we're assuming and enforcing one single way/structure of testing. That isn't the case. And different users and different frameworks might test in different ways, so that one structure doesn't make sense to not be changeable.