search

FortuneN / FineCodeCoverage

Visualize unit test code coverage easily for free in Visual Studio Community Edition (and other editions too)
https://marketplace.visualstudio.com/items?itemName=FortuneNgwenya.FineCodeCoverage
Other
516 stars 39 forks source link

Tests are running twice #342

Closed sodablue closed 1 year ago

sodablue commented 1 year ago

In Visual Studio 2022, latest 17.6.5, noticed if I selected a single unit test in Test Explorer and hit Run, it would actually run all tests in the test class. If I selected the class, it would run every test in the class twice. Adding debug.writeline, or break points and debugging doesn't show up, doesn't halt on the code. The only reason I noticed they're running is because these were integration tests hitting a database and I saw the extra records being created, otherwise it's likely not visible without some test side effect.

Digging into this, I found this tip over on SO: https://stackoverflow.com/questions/70231309/xunit-nunit-msunit-all-run-all-tests-twice-vs2022

And I followed their advice. I first tried disabling FineCodeCoverage extension, and that solved the problem. I took the advice of waistcoat1971 and changed the RunMsCodeCoverage setting and that also solved the problem.

I haven't dug into the FineCodeCoverage codebase to understand why this is happening. Again it's only reproducible if your tests have some sort of side effect.

tonyhallett commented 1 year ago

This is documented in the readme and is by design. RunMsCodeCoverage is the recommended setting.

sodablue commented 1 year ago

Can this be the default setting? Why would you want the extension to be broken by default?

tonyhallett commented 1 year ago

Can this be the default setting?

FCC introduced Ms code coverage as a coverage option when it became available, this was some time after FCC had been using Coverlet and OpenCover. It was not set as default as the code needed to be tested by users of the extension. It has been running successsfully but there is one outstanding issue that people may encounter that should be fixed before it becomes the default.

The old style coverage has no issues. Yes it run tests twice and copies dll's to a different directory and thus is not suited to particular tests, in particular, UI tests. It is not broken, it performs as documented and

Tests are running twice

has already been addressed in github issues on more than one occasion. If the behaviour is not desirable then you can switch to ms code coverage.

Note though that old style coverage does provide functionality that you cannot get with ms code coverage - which was the original use case of the extension when @FortuneN developed it. With old style you can go about your testing duties with FCC generating a report in the background for all tests in the test projects that you run. When you want the complete coverage it will just be there. Ms code coverage on the other hand will only run tests and consequently coverage for the tests that you have selected.