Applying [Explicit] and [Ignore] Attributes to Test Class with Custom Category Ignores All Tests in Category

[dhrumil0902]

Description When applying the [Explicit] attribute and [Ignore(...)] attribute to a test class that has a custom category attribute, all classes within that category are ignored when running dotnet test while filtering by that category. Therefore no tests are ran. Please let me know if I am doing something wrong.

Steps to Reproduce

1. Use the example code below, or create similar classes with those attributes.
2. Run dotnet test --filter TestCategory=MyCustomCategory on the test project.

Example Code

using NUnit.Framework;

namespace TestProject1
{
    [TestFixture, Category("MyCustomCategory")]
    [Explicit]
    [Ignore("Ignoring")]
    public class MyTestClass
    {
        [Test]
        public void MyTest()
        {
            Assert.Pass();
        }
    }

    [TestFixture, Category("MyCustomCategory")]
    [Explicit]
    public class AnotherTestClass
    {
        [Test]
        public void AnotherTest()
        {
            Assert.Pass();
        }
    }
}

Expected Behavior Only the specific test class with the [Ignore(...)] attribute should be ignored. Other test classes in the same category should execute as expected.

  Determining projects to restore...
  All projects are up-to-date for restore.
  TestProject1 -> C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll
Test run for C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Skipped MyTest [< 1 ms]

Passed!  - Failed:     0, Passed:     1, Skipped:     1, Total:     2, Duration: 39 ms - TestProject1.dll (net8.0)

Actual Behavior All test classes within the same custom category are ignored, causing all tests in that category to be skipped.

  Determining projects to restore...
  All projects are up-to-date for restore.
  TestProject1 -> C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll
Test run for C:\Users\dhpatel\source\repos\TestProject1\bin\Debug\net8.0\TestProject1.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Skipped AnotherTest [5 ms]
  Skipped MyTest [< 1 ms]

Skipped! - Failed:     0, Passed:     0, Skipped:     1, Total:     1, Duration: 9 ms - TestProject1.dll (net8.0)

CsProj File

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.10.0" />
      <PackageReference Include="NUnit" Version="4.1.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.5.0" />
      <PackageReference Include="NunitXml.TestLogger " Version="3.1.20" />
  </ItemGroup>

</Project>

[dhrumil0902]

@OsirisTerje pls take a look if possible =)

[OsirisTerje]

I just ran these in the Test explorer, and add the category as a trait,and using Run All they are not executed for different reasons. The MyTest because it is Ignored, the AnotherTest because it is Explicit, but it should have run.

You see the different icons for them:

If I remove the Ignore attribute, they do run both of them, as they should.

So it seems the Ignore does mess up something here. Nice catch!

If I add a third test class, with the same category, it does execute, but the two others doesnt, for the same different reasons. So, yes, this is a bug.

Repro uploaded to https://github.com/nunit/nunit.issues/tree/main/Issue4736

@manfred-brands Want to have a look ?

[dhrumil0902]

@manfred-brands any updates? @OsirisTerje I can take a look at this bug if you can point me in the right direction. I also saw on your profile that you are looking for contributors. Where and how can I start contributing?

[OsirisTerje]

The best way to debug this is to take the repro above, add that to a solution which also includes the framework. So clone both into their respective local repos, and let the solution span both projects. Then it is easy to get into debugging them, and see what happens with the filter.
Note, it may be a bug in the adapter too, but currently it does look like the error is in the framework. To make sure it is in the framework and not adapter, it could be wise to run the repro using NUnitLite.

About contributing: Yes, we do want more contributors. Check out the contributing guidelines and the NUnit.org website. You should also check the NUnit governance documentation which explains the roles and project structures. Also check the docs, which may be helpful.

[dhrumil0902]

The bug is not there when executing tests with NUnitLite or nunit-console, so looks like the problem is in the adapter.

[OsirisTerje]

Ok, thanks! I'll move this issue to the adapter.
To debug into the adapter follow the steps outlined here: https://hermit.no/debugging-the-nunit3testadapter-take-2/ But do build a debug version, it makes it easier for going in there.

[dhrumil0902]

@OsirisTerje After debugging the code in the package, I have an idea of where this bug is coming from, but I have multiple questions. Is there a better medium to ask questions and/or ask for help?

[OsirisTerje]

You can join https://nunit.slack.com. Here is an invite link https://join.slack.com/t/nunit/shared_invite/zt-2o4x0nfoh-llr_V6GO5ZI4TfCXqjUUAw

[amaltinsky]

@OsirisTerje Hi! Any updates on this bug? It's causing us some trouble, preventing us from using the [Ignore] attribute.

[OsirisTerje]

@dhrumil0902 started doing something on this issue a few weeks back. Not sure where it is now, but he might answer.