[BUG] Hooks are ignored if the class in which they are defined lives in a different assembly than the tests themselves.

[Xen0byte]

The example use case here would be a class library that sets up things for multiple test projects in a consistent way so that those test projects can reference this one library and run consistently across the board. MSTest, NUnit, and xUnit all support this scenario so, hopefully, this is indeed a bug, and not the intended behaviour.

Repro Steps:

1. create 2 projects; project A contains some tests, while project B contains a class with some hooks (e.g. before/after test)
2. make sure project A references project B, and make the class containing the tests inherit from the class in project B
3. put some breakpoints on the hooks and observe that they never get hit

[thomhurst]

@Xen0byte Can you try on version 0.1.773 ?

I'm successfully hitting breakpoints in the hook class:

I've added a test too to verify that the hooks were hit: https://github.com/thomhurst/TUnit/blob/main/TUnit.TestProject/BasicTests_HooksFromLibrary.cs

[thomhurst]

The above was Rider ^

And it's hitting is Visual Studio too:

[Xen0byte]

Hi @thomhurst, thank you for getting back to me so quickly. After looking at your unit tests, I've done more digging and finally discovered what the problem was. I was using TUnit.Core in my library, and it turns out that I needed TUnit.Engine instead. After sorting this out, however, I'm now getting test discovery exceptions in my NUnit and xUnit projects, so at this point it's probably worth providing additional context around what I'm building so you can tell me if my potentially very specific scenario can be covered. Essentially I'm working on a library to unify unit testing across my entire organisation (probably around 300-ish developers and QA people who also write tests). This library, amongst other things, sets up the exact same before/after assembly/fixture/test hooks for MSTest, NUnit, xUnit, Reqnroll (SpecFlow successor), and now I'm trying to add TUnit to that list too, so that everybody across the organisation can get the exact same experience regardless of which unit testing framework their team might prefer, and I should probably note that it's not only about preference, but also about being able to retro-fit this library into pre-existing test projects.

So... to make a long story short, TUnit doesn't seem to play well with the others in this potentially very specific scenario.

For instance ...

    <ItemGroup>
        <PackageReference Include="Microsoft.Extensions.Configuration" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="9.0.0-rc.1.24431.7" />
        <PackageReference Include="Microsoft.Playwright" Version="1.47.0" />
        <PackageReference Include="MSTest.TestFramework" Version="3.6.0" />
        <PackageReference Include="NUnit" Version="4.2.2" />
        <PackageReference Include="Reqnroll" Version="2.1.0" />
        <PackageReference Include="TUnit.Core" Version="0.1.773" />
        <PackageReference Include="xunit.extensibility.core" Version="2.9.0" />
    </ItemGroup>

... works fine, but the TUnit hooks don't get hit, and when I fix this by replacing ...

<PackageReference Include="TUnit.Core" Version="0.1.773" />

... with ...

<PackageReference Include="TUnit.Engine" Version="0.1.773" />

... I start getting a bunch of errors on solution build, along the lines of ...

Unable to connect to testing platform runner process 'path\to\XXX.Core.Tests.NUnit.exe'.

or

Unable to connect to testing platform runner process 'path\to\XXX.Core.Tests.xUnit.exe'.

... where these binaries represent the multiple test projects that I'm using to test the library.

Any thoughts that you might have on this would be greatly appreciated. I would very much like to promote TUnit internally as the preferred framework of choice, but I can only do that if I can integrate it successfully.

[thomhurst]

Ah yes. I need to do some refactoring as I'd like to have it so that you only need to import the "Core" package for libraries. I'll raise an issue so I can hopefully sort it, but as you've realised for now you can just use the standard TUnit package.

I believe your error is due to mixing the old VSTest and the new Microsoft.Testing.Platform. They're essentially the old MS runner and the new MS runner, and I think you basically have to say which you want to use.

When you install TUnit, it implicitly sets some project properties for you so you don't have to do it manually. So I think what's happening is it's basically telling your project to run in the new Microsoft.Testing.Platform mode, but your other test frameworks don't currently support that, and hence you get that error.

You should be able to explicitly set these properties in your csproj to disable this, but you'll need to still enable it for TUnit.

The easiest way to get this running is probably by having an environment variable set for when you want to use TUnit, and then using property conditions in your csproj.

So do something like this:

<PropertyGroup Condition="'$(EnableTUnit)' != 'true'">
        <IsTestingPlatformApplication>false</IsTestingPlatformApplication>
        <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport>
</PropertyGroup>

And then when you do want to use TUnit, just set the environment variable EnableTUnit=true

Or on the CLI pass in --property:EnableTUnit=true

Give that a go if you can and let me know if it helps!

Edit: See comment below

[thomhurst]

Oh actually looking at your screenshot, you have separate test projects for each framework. In the non-TUnit ones just add to their csprojs

<PropertyGroup>
        <IsTestingPlatformApplication>false</IsTestingPlatformApplication>
        <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport>
</PropertyGroup>

[Xen0byte]

Indeed, your analysis of the problem is spot on, however the suggested solution doesn't seem to be doing the trick for me. I've even tried to specify the runners explicitly, e.g. ...

    <PropertyGroup>
        <VSTestRunner>VSTest</VSTestRunner>
        <IsTestingPlatformApplication>false</IsTestingPlatformApplication>
        <TestingPlatformDotnetTestSupport>false</TestingPlatformDotnetTestSupport>
    </PropertyGroup>

... and, respectively ...

    <PropertyGroup>
        <TestingPlatformRunner>Microsoft.Testing.Platform</TestingPlatformRunner>
        <IsTestingPlatformApplication>true</IsTestingPlatformApplication>
        <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
    </PropertyGroup>

I also tried taking a different approach, by upgrading all the other projects to Microsoft.Testing.Platform and that seems to kind of solve the problem, but all the dependencies required for the upgrade are fairly unstable, so I don't think I can do that just yet.

On top of that, after the upgrade, I keep getting warnings left and right, e.g. The type 'SelfRegisteredExtensions' in '...\XXX.Core.Tests.TUnit\obj\Debug\net9.0\AutoRegisteredExtensions.cs' conflicts with the imported type 'SelfRegisteredExtensions' in 'XXX.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'. Using the type defined in '...\XXX.Core.Tests.TUnit\obj\Debug\net9.0\AutoRegisteredExtensions.cs'. or Method 'TestingPlatformEntryPoint.Main(string[])' will not be used as an entry point because a synchronous entry point 'AutoGeneratedEntryPoint.Main(string[])' was found. which can be resolved with <GenerateTestingPlatformEntryPoint>false</GenerateTestingPlatformEntryPoint>.

I think that for the time being I'll wait until you manage to find some time for refactoring the bits that allow libraries to only require TUnit.Core and after that attempt an upgrade to Microsoft.Testing.Platform across the board again.

Many thanks for the help and the quick responses, and congratulations for making it on Nick Chapsas' vlog today. I think a lot of people are really excited about what you're building here and I can't wait to see what's coming next, and even though for this one project I'll put TUnit on the backburner for a little while until I can get everything to play nice together, TUnit is already my framework of choice and I will continue to try to break it in many interesting ways.

[thomhurst]

Thanks for the update @Xen0byte ! I'll try and work on sorting out the core package soon to hopefully unblock you. In regards to upgrading your other projects to the testing platform, this won't work. They will need new code changes to interact with the new testing protocol. NUnit and xUnit have been working on it, but I don't know if it'll work for your existing projects. I think you'll have to stick to vstest for now, but I'll try to unblock the core package problems still which I think should hopefully still sort your problems!