Missing and/or misplaced tests in the Test Explorer when using multiple test projects

BalassaMarton commented 8 years ago

After following Rob Prouse's instructions on using NUnit with .NET Core projects, I've discovered that tests are missing or misplaced in the test explorer when there are multiple test projects in the solution.

Steps to reproduce

(see the attached solution 'NUnitMultipleTestProjects' for details)

Create new Class library (.NET Core) project
Add new solution folder 'test'
Add new Class library (.NET Core) project to the test folder (TestProject1)
Modify project.json to enable NUnit
Add a test class to the project (TestClass1)
Add test methods to the project (Test1 and Test2)
Repeat steps 3-6 to create another project with a test class and test methods (TestProject2, TestClass2, Test3, Test4)
Save and build the solution to discover tests.
Expected behavior

In the Test Explorer tree view, you should see the following structure: TestClass1 -- Test1 -- Test2 TestClass2 -- Test3 -- Test4

Actual behavior

The tree view has a single TestClass1 node, and the children of this node depend on what files you've edited before hitting Build. Sometimes it has the test methods of TestClass2, sometimes mixed, but TestClass2 never appears in the tree view, and you cannot run tests that are not listed.

nunitmultipletestprojects -- figure 1

Additional notes and workaround

UPDATE Turns out the following workaround does not work. Tests are misplaced even when there is only a single test project with multiple test fixtures.

After experimenting for a couple of hours, I've discovered that the problem occurs only when there are multiple test projects in the solution. The tree view will contain class names from the first project, and under those class names you will find the test method names from the project you modified last.

nunitmultipletestprojects -- figure 2 That's right, you modify a test method, hit Build, and find the tests from that project listed under the classes from the first project. When you place all your tests in a single project, the test classes and methods are displayed correctly. Also, I'm not entirely sure that this is not a bug in the Test Explorer itself, so I will try out the same project structure with different test frameworks later.

Prepared solutions for reproducing the issues: NUnitMultipleTestProjects.zip NUnitMultipleTestProjectsMultipleTestClasses.zip NUnitSingleTestProjectMultipleTestClasses.zip

rprouse commented 8 years ago

I have seen the same thing. For me, restarting Visual Studio fixed the problem, so I assumed it was a bug in the .NET Core support within Visual Studio. If you get a chance to test other frameworks, I would appreciate the help.

BalassaMarton commented 8 years ago

I've just repeated the steps with xUnit.net, based on this article, and the tests appeared as expected, so I think it is a NUnit issue after all.

rprouse commented 8 years ago

Thanks for the help. We will look into it.

tsublette commented 8 years ago

I am experiencing the same thing. I have three class libraries that I publish to a myget repo. Each has their own test project. After porting the libraries to .net core with the instructions above, I get all the tests from the last project I added and some random number of tests from the other projects. As a work around, I can use command line dotnet test in the directory of each test project. However, it would be great to be able to run all in one place.

matt-hosking commented 8 years ago

I am having issues with the same behaviour. I noticed that out of 3 test projects, I was only seeing one in the output. But when I look at the output of the first test, it also contains the output from running the first test in the other two projects. Adding a second test to each project shows them similiarly grouped. As noted above, the project and test that is shown seems to vary, but the tests at the same position across all projects are grouped.

rprouse commented 8 years ago

Thanks for the info, that might help us track down the issue.

peter-dangelo commented 8 years ago

i'm having the same issue

tsublette commented 7 years ago

Bump... Any update on this issue. Fixing would make a world of difference working in the IDE.

-Thanks

rprouse commented 7 years ago

Sorry no progress yet.

dougkpowers commented 7 years ago

Perhaps the following information will help track down this issue...

I have two test projects-- project A one with 125 tests and project B with 1 test. Running the tests from the command line works, and all tests pass (in both projects).
The output in VS shows that 125 tests were correctly discovered in Project A and that 1 test was discovered in project B.
The Test explorer shows 124 tests in Project and and 1 test in project B.
When running within visual studio, it shows 124 tests passing in project A and 1 test failing in project B.
I removed the test from project B so that it now contains 0 tests. I run the tests again, and it still says it is running 125 tests. In test explorer, it still shows 124 tests from Project A and 1 test from Project B (even though project B doesn't have a test anymore). It shows all tests from both projects as passing.

jplebre commented 7 years ago

Tonight was the first time I've seen this. I've started copying tests and giving them an incremental number at the end of the signature, but it seems the explorer randomly chooses to either drop one of the tests from either project. I also had it dropping testdropped1 or testdropped2 from the same project randomly.

jplebre commented 7 years ago

Some picks: 2x VerifyConnection are in the acceptance test project 3x Feedback form and 2x QueryReturns tests are in the test project

Note, no query return showed:

Now I add one more QueryReturn: Interestingly enough, the feedbackform test changed state. Also, now none of the tests are shown from the acceptance test project (VerifyConnection1 and 2)

Lets add another QueryReturn: Yet another previously run test went to not run state.... hmmm.... All QueryReturns tests are showing up

Lets add a VerifyConnection (number 3) this time: We now have 3x 'FeedbackForm', 4x'QueryReturn' and 3x 'VerifyConnection' How's that for a Friday evening puzzle? Look how it arbitrarily picked QueryReturn4 to remain

As pointed out above, console runs them fine.

Restarting VS2015 does not fix this for me unfortunately :(

dougkpowers commented 7 years ago

Pull Request #82 submitted as a proposed fix for this issue.

Visual studio runs the test discovery process for each project in the solution. During each run, the "Id" attribute on the Xml node for the test starts at 1001. Therefore, the Ids for all tests in a solution are not unique (since the id for the first test in each project will be 1001).

The changes in this pull request use the fullname attribute to generate a Guid. Because the fullname includes the entire namespace of the class (plus the method), I think this would be sufficient to generate a unique Guid. If for some reason the fullname is not available on the test, it falls back to using the "Id" attribute to generate the Guid.

For those in need of a quick fix (as I was), pull #82 into your local branch, create a nuget package (dotnet pack), reference this nuget package, and report back your results.

tsublette commented 7 years ago

Thank you so much for fixing this and getting it merged. What are the plans / timeline for releasing this to Nuget? If it is a ways out, I can fork the project and pull it into my working project temporarily.

rprouse commented 7 years ago

I am hoping to do a release this weekend. I am just waiting on some additional fixes in the agent project that this depends on.

byjlw commented 7 years ago

I'm having this issue currently I'm on 3.7. Was this ever released?

nunit / dotnet-test-nunit