search

microsoft / vstest

Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
MIT License
870 stars 316 forks source link

Traits (Category) should differentiate test cases even when they have same data name #5097

Open PeterHevesi opened 2 weeks ago

PeterHevesi commented 2 weeks ago

Hi there, I have found, that only because same dataset has the very same name as another dataset coming from different datasource, it is not discovered in VS Test Explorer. But it should, because by putting different Category (from NUnit), I already want to clearly say, that that datasource is actually different. Even when I use Traits to group my tests, these tests are never discovered when having the same name. I have made simple NUnit Test project with this simple file, which should absolutely clearly illustrate what I mean:

https://gist.github.com/PeterHevesi/aa1392c27000985c13c35dff9653f83a

nohwnd commented 2 weeks ago

Looking at your code I don't see anything vstest specific, this looks like a problem specific to NUnit. Have you tried filing the issue in Nunit? https://github.com/nunit/nunit

PeterHevesi commented 1 week ago

Well I have created an issue in nunit3-vs-adapter, here: https://github.com/nunit/nunit3-vs-adapter/issues/1182

And they told me, that it is not on their side, but it is how Test Explorer is identifying tests by FQN. But couldn't category be put into that FQN? So just by having different category (traits) on a testcasesource, it could have the very same name and would be properly found as different test cases?

Don't know why Category is not put into that FQN to differentiate tests, it doesn't seems right to me, and it makes working with testCaseSources harder!

nohwnd commented 1 week ago

This would be a big change, but you can file an issue to TestExplorer by pressing the feedback button in VS. It would also require nunit to add that information to the fully qualified name so it becames distinct.

PeterHevesi commented 1 week ago

Ok thanks, I have filed an issue there :) I hope they will fix this. It is only very logical to work like I am proposing :) https://developercommunity.visualstudio.com/t/Traits-Category-should-differentiate-t/10685328?port=1025&fsid=ff50b090-f972-4576-800e-7eb71583d2e2

PeterHevesi commented 1 week ago

This work could also try to solve another problem, that is currently there. We are discussing it with @OsirisTerje at this issue: https://github.com/nunit/nunit3-vs-adapter/issues/1182#issuecomment-2176405468

As I said, it could only be very logical to even add another attributes besides category, as follows:

  1. CategoryGroup
  2. Category
  3. Subcategory
  4. Tag

or even more attributes, if someone would need them.

All of these would be part of FQN, to differentiate tests from different datasources, even when they have same dataname. Each attribute could be then used separately to categorize by some grouping, for example in my scenario, we have different website version for each of our customer, I would use Category for grouping by website version. Then I would also like to group by let's say browser those tests should be run on. I would use Subcategory for that.

And you could of course group by all of these 4 attributes independently in Test Explorer !!!

PeterHevesi commented 4 days ago

Hi @nohwnd, so what do you think about this? Is there any chance to make category to be part of FQN? It won't change the current behavior in any way, as I understand it. It would only fix the problem I described at the beginning

PS: If you agree with me, that this change won't do any harm to existing behavior, and everything would work exactly the same after this change, would you please upvote this feature request? https://developercommunity.visualstudio.com/t/Traits-Category-should-differentiate-t/10685328?