search

FortuneN / FineCodeCoverage

Visualize unit test code coverage easily for free in Visual Studio Community Edition (and other editions too)
https://marketplace.visualstudio.com/items?itemName=FortuneNgwenya.FineCodeCoverage
Other
506 stars 38 forks source link

Tests runs sevaral times #418

Closed AndersMalmgren closed 2 months ago

AndersMalmgren commented 3 months ago

I noticed a stranger behaviour in our Integration tests. Several test-databases were created even if I just ran one integration test. It turns out after alot of digging around that Fine code coverage runs the test several times. North of 50 times for one test.

Is this by design?

AndersMalmgren commented 3 months ago

Also I though Fine code coverage only did its thing when the window was focused. But it does stuff under the hood even if the window is not infocus or opened. Funny thing is when I do open the window. It does not use the background data it collected, you need to run the tests again for it to capture the run.

tonyhallett commented 3 months ago

I noticed a stranger behaviour in our Integration tests. Several test-databases were created even if I just ran one integration test. It turns out after alot of digging around that Fine code coverage runs the test several times. North of 50 times for one test.

Is this by design?

FCC will, unless RunMsCodeCoverage Yes, run all tests in the test assembly after the test explorer has finished running the tests that you have selected. It does not run a test several times.

AndersMalmgren commented 3 months ago

It does run several times. I run one test and I can see that it runs test init several times because it creates multiple databases.

tonyhallett commented 3 months ago

No. It runs all tests in the test assembly once regardless of the test/s that you have run in the test explorer. If you set RunMsCodeCoverage Yes then FCC will not run any tests of its own. Coverage will be for the tests that you have selected in the test explorer.

AndersMalmgren commented 3 months ago

Thats not what I have observed. Try this repro.

https://github.com/AndersMalmgren/FccRepro

Run only one of the tests. It will run all tests in suite.

edit: it saves files under c:\temp\fcctest edit2: RunMsCodeCoverage Yes stops it from generating in teh background. But it also stops it from showing anything in the FCC UI when I run the tests with teh window open

tonyhallett commented 3 months ago

Yes FCC runs all the tests, but once.

AndersMalmgren commented 3 months ago

Yes FCC runs all the tests, but once.

If you like us have 200+ tests that is not ideal. If RunMsCodeCoverage Yes is fixed so it works when window is in focus that is an OK solution though. Now it just displays an empty summary window. Though hot spots seems to work.

tonyhallett commented 3 months ago

The behaviour is documented in the Readme and there have been multiple closed duplicate issues on the described behaviour.

As for RunMsCodeCoverage being fixed - I don't believe it is broken. What are your settings ? Have you set IncludeReferencedProjects ? In the bin directory you will find the generated runsettings. Is it correct ?

AndersMalmgren commented 3 months ago

I guess disabling the extension and enable it when you want to check coverage works but its not ideal.

tonyhallett commented 3 months ago

I guess disabling the extension and enable it when you want to check coverage works but its not ideal.

Or you could RunMsCodeCoverage Yes with generated runsettings or supply your own.

AndersMalmgren commented 3 months ago

I guess disabling the extension and enable it when you want to check coverage works but its not ideal.

Or you could RunMsCodeCoverage Yes with generated runsettings or supply your own.

RunMsCodeCoverage Yes has a bug, if its set to true summary pane does not output anything even though I see in the log that it have run correctly

tonyhallett commented 3 months ago

RunMsCodeCoverage Yes works on my system and also with other users of the extension. So it is either a bug that has yet to surface, possibly due to the specifics of your project or it just isn't collecting because it has not been configured to collect.

If you can provide the information I requested then it can be determined.

badams-eso commented 2 months ago

@AndersMalmgren @tonyhallett I am also experiencing this issue and have done a bit of digging which can hopefully prove useful for you.

FCC with RunMsCodeCoverage use to work amazingly and now it does not for me. I'm not getting NOTHING in the coverage panel as Anders describes, but it seems to only display coverage for unit test projects. Looking at the raw coverage XML that is output there is only coverage for the unit test projects. So FCC is displaying this correctly, but I wonder is the runsettings file it's creating somehow causing ms code coverage to not collect for all projects?

Experimenting a bit further just now, the generated runsettings file specifies a single module to include which is the test dll. When I copy this runsettings file elsewhere (so FCC doesn't delete it) and then set it as my solution-wide runsettings with no modules specifically included it works again like a charm.