search

FortuneN / FineCodeCoverage

Visualize unit test code coverage easily for free in Visual Studio Community Edition (and other editions too)
https://marketplace.visualstudio.com/items?itemName=FortuneNgwenya.FineCodeCoverage
Other
527 stars 40 forks source link

Unknown Issue #147

Closed madmoravian closed 2 years ago

madmoravian commented 3 years ago

Installed product versions

Description

Replace this text with a short description

Steps to recreate

  1. Ran the tests
  2. Examine FCC log
  3. Nothing but 0 percentages in the coverage report

Current behavior

Fine Code Coverage : Run Coverage Tool (GatewayDataServices.IntegrationTests) Failed System.IndexOutOfRangeException: Index was outside the bounds of the array. at FineCodeCoverage.Core.Utilities.DotNetToolListParser.<>c.b__0_0(String l) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Utilities\DotNet\DotNetToolListParser.cs:line 18 at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at FineCodeCoverage.Core.Utilities.DotNetToolListParser.Parse(String output) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Utilities\DotNet\DotNetToolListParser.cs:line 16 at FineCodeCoverage.Engine.Coverlet.DotNetToolListCoverlet.ExecuteAndParse(Func`2 execution) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Coverlet\Console\DotNetToolListCoverlet.cs:line 34 at FineCodeCoverage.Engine.Coverlet.DotNetToolListCoverlet.Global() in C:\projects\finecodecoverage\FineCodeCoverage\Core\Coverlet\Console\DotNetToolListCoverlet.cs:line 50 at FineCodeCoverage.Engine.Coverlet.CoverletConsoleDotnetToolsGlobalExecutor.GetRequest(ICoverageProject coverageProject, String coverletSettings) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Coverlet\Console\CoverletConsoleDotnetToolsGlobalExecutor.cs:line 26 at FineCodeCoverage.Engine.Coverlet.CoverletConsoleUtil.GetExecuteRequest(ICoverageProject project, String coverletSettings) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Coverlet\Console\CoverletConsoleUtil.cs:line 54 at FineCodeCoverage.Engine.Coverlet.CoverletConsoleUtil.d8.MoveNext() in C:\projects\finecodecoverage\FineCodeCoverage\Core\Coverlet\Console\CoverletConsoleUtil.cs:line 127 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FineCodeCoverage.Engine.Model.CoverageProject.d62.MoveNext() in C:\projects\finecodecoverage\FineCodeCoverage\Core\Model\CoverageProject.cs:line 343

I have no idea if the exception is related to the fact that no results are generated.

tonyhallett commented 3 years ago

Will have a look asap

tonyhallett commented 3 years ago

You have set the visual studio option CoverletConsoleGlobal. The error, rather unhelpful that it is, could be an error on my behalf or that you do not have a globally installed coverlet console dotnet tool.

If you have installed the tool globally then please bear with me and I will look into it tomorrow.

In the mean time you could provide the path to the global tool with the CoverletConsoleCustomPath and if that does not work then use FCC's installed copy of coverlet console.

Please let me know if you have mistakenly selected the global tool and if not if you had success with the alternatives.

tonyhallett commented 3 years ago

From the discussion we had before - https://github.com/FortuneN/FineCodeCoverage/discussions/112#discussioncomment-553153

Yes, I've updated to 3.0.3 and I think other folks are using the console that comes with the installation.

It would suggest that you have globally installed.

Note that FCC itself uses v3.0.3.

You may have the data collector driver as an alternative solution to this problem.

Fine Code Coverage will use the Data Collector driver under two circumstances : You are testing with runsettings that contains the coverlet collector ( and not disabled) You set the UseDataCollector project property

madmoravian commented 3 years ago

I've turned off the global option and still get zeros across the board for coverage results. That exception has gone away.

I did also see this error:

Fine Code Coverage : Coverlet Run (GatewayDataServices.UnitTests) Starting test execution, please wait... Logging Vstest Diagnostics in file: C:\Users\wzrubek\Documents\Gateway\gatewaydataservices\GatewayDataServices.UnitTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output\diagnostics.log A total of 1 test files matched the specified pattern. Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown. Passed! - Failed: 0, Passed: 3168, Skipped: 0, Total: 3168, Duration: 10 m 37 s - GatewayDataServices.UnitTests.dll (net5.0)

tonyhallett commented 3 years ago

Did you have the global tool installed ? It determines the amount of work I need to do regarding the exception.

Does the diagnostic file suggest anything ?

tonyhallett commented 3 years ago

Is that the full FCC log ?

If FCC has completed and you have all zeros then that suggests a known coverlet console issue and as advised you should use the data collector driver instead.

madmoravian commented 3 years ago

The global tool is installed.

The diagnostics file shows:

TpTrace Information: 0 : 20292, 1, 2021/05/19, 17:47:08.780, 199107504108, vstest.console.dll, Failed to create newly implemented Fakes Configurator. Reason:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=16.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. File name: 'Microsoft.VisualStudio.TestPlatform.Fakes, Version=16.0.0.0, Culture=neutral, PublicKeyToken=null' at System.Reflection.RuntimeAssembly.InternalLoad(ObjectHandleOnStack assemblyName, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Boolean throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly) at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, RuntimeAssembly requestingAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext) at System.Reflection.Assembly.Load(AssemblyName assemblyRef) at Microsoft.VisualStudio.TestPlatform.Common.Utilities.FakesUtilities.TryGetFakesCrossPlatformDataCollectorConfigurator() TpTrace Verbose: 0 : 20292, 1, 2021/05/19, 17:47:08.782, 199107528685, vstest.console.dll, TestPluginCache.DiscoverTestExtensions: finding test extensions in assemblies ends with: TestLogger.dll TPluginInfo: Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.TestLoggerPluginInformation TExtension: Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestLogger

Other than the exception for not finding AsyncInterfaces, which seems to show up whether it's working or not, this was the only exception I saw in the diagnostics.

tonyhallett commented 3 years ago

Did you show the full FCC log ?

If FCC has completed and you have all zeros then that suggests a known coverlet console issue and as advised you should use the data collector driver instead.

tonyhallett commented 3 years ago

Just so you are aware FCC copies your test project output to a sub directory. Perhaps your paths are incorrect.

madmoravian commented 3 years ago

Yes, FCC complete, apparently without issue.

I'll give the DataCollector Driver solution a try. I've added that property to all of my projects just to be safe.

madmoravian commented 3 years ago

After almost an hour, FCC has not run to completion.

tonyhallett commented 3 years ago

Unless configured otherwise FCC will run after your tests have finished so I'd expect at least double test execution time but from what you originally posted Duration: 10 m 37 s an hour seems to long.

Without being able to debug your project there is not a lot that I can do from here.

You could run against coverlet manually outside of FCC and see if you get coverage results that way and then we would know that it is likely that there is a bug with FCC.

tonyhallett commented 3 years ago

I have updated FCC to not throw the error when the CoverletConsoleGlobal option fails and to output the text that we attempt to parse. Please can you install this vsix when you get a moment, selecting the option again. If you could post what is logged prior to "Unable to use Coverlet console global tool" then I can understand why the parsing failed.

madmoravian commented 3 years ago

With Global and Local coverlet turned off, I spotted this error in the FCC output after installing the vsix

Fine Code Coverage : ================================== ERROR ==================================

Fine Code Coverage : System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString() at System.IO.StreamReader.ReadToEnd() at System.IO.File.InternalReadAllText(String path, Encoding encoding, Boolean checkHost) at System.IO.File.ReadAllText(String path) at FineCodeCoverage.Core.Utilities.FileUtil.ReadAllText(String path) in C:\projects\finecodecoverage\FineCodeCoverage\Core\Utilities\FileUtil.cs:line 43 at FineCodeCoverage.Engine.ReportGenerator.ReportGeneratorUtil.d14.MoveNext() in C:\projects\finecodecoverage\FineCodeCoverage\Core\ReportGenerator\ReportGeneratorUtil.cs:line 143 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FineCodeCoverage.Engine.FCCEngine.d45.MoveNext() in C:\projects\finecodecoverage\FineCodeCoverage\Core\FCCEngine.cs:line 167 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FineCodeCoverage.Engine.FCCEngine.<>c__DisplayClass49_0.<b__0>d.MoveNext() in C:\projects\finecodecoverage\FineCodeCoverage\Core\FCCEngine.cs:line 265

madmoravian commented 3 years ago

With Global turned on:

Fine Code Coverage : Coverlet Run (GatewayDataServices.UnitTests) Starting test execution, please wait... Logging Vstest Diagnostics in file: C:\Users\wzrubek\Documents\Gateway\gatewaydataservices\GatewayDataServices.UnitTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output\diagnostics.log A total of 1 test files matched the specified pattern. Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown. Passed! - Failed: 0, Passed: 3243, Skipped: 0, Total: 3243, Duration: 11 m 6 s - GatewayDataServices.UnitTests.dll (net5.0)

tonyhallett commented 3 years ago

With Global turned on:

That's bizarre that there is no message re the issue with the global tool you encountered before.

As per the out of memory - I introduced a change recently that read the generated files into memory. I will revert tomorrow and then try to address the issue at hand.

tonyhallett commented 3 years ago

Please try this vsix and report the full fcc output.

tonyhallett commented 3 years ago

Aplogies if you have installed the vsix above. This vsix has slightly better error output.

tonyhallett commented 3 years ago

When you get the opportunity can you remove the <UseDataCollector/> and run with the CoverletConsoleGlobal option so I can see why the "dotnet tool list --global" parsing failed

madmoravian commented 3 years ago

I've loaded the latest vsix, removed the UseDataCollector from all of my projects, and turned on CoverletConsoleGlobal.

There is no occurrence of DotNetToolListParser in the FCC output or the diagnostics log.

The only weird thing I saw on runs with either CoverletConsoleGlobal true or false was:

Fine Code Coverage : Coverlet Run (GatewayDataServices.UnitTests) Starting test execution, please wait... Logging Vstest Diagnostics in file: C:\Users\wzrubek\Documents\Gateway\gatewaydataservices\GatewayDataServices.UnitTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output\diagnostics.log A total of 1 test files matched the specified pattern. Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown.Exception of type 'System.Exception' was thrown. Passed! - Failed: 0, Passed: 3243, Skipped: 0, Total: 3243, Duration: 10 m 18 s - GatewayDataServices.UnitTests.dll (net5.0)

tonyhallett commented 3 years ago

My understanding is that the exception messages are ( expected ) exceptions thrown in your tests.

I don't understand the logging you are displaying as this is the only code that logs "Coverlet Run....."

public async Task<bool> RunAsync(ICoverageProject project, bool throwError = false)
        {
            var title = $"Coverlet Run ({project.ProjectName})";

            var coverletSettings = GetCoverletSettings(project);

            logger.Log($"{title} Arguments {Environment.NewLine}{string.Join($"{Environment.NewLine}", coverletSettings)}");

as such I can only assume that you are not showing the full FCC output.

What vs options ( are you running in parallel ? ) / project options have you set ? Are you using run settings ?

This vsix adds an additional error logging line.

If nothing else surfaces I suggest that you build FCC and debug or you give me rights to your project and I will debug.

tonyhallett commented 3 years ago

Or run coverage with Coverlet outside of FCC to determine if there is an issue there.

In case you were not aware. If you run tests to start coverage then run further tests before FCC has completed the process will begin again.

Final thought. Perhaps for your test suite it does take a very long time for Coverlet to complete, for ReportGenerator to provide the two required reports and for FCC to tranform the generated html.

madmoravian commented 3 years ago

The vs options we have: Global: False, Local: False, RunSettnigsOnly: True, RunInParallel: False It takes about 10 minutes this morning. I've attached (I hope ) a complete run log. Unfortunately, I'm unable to give outside parties access to the projects. FCCoutput.log

tonyhallett commented 3 years ago

Ok so FCC completes although with 0% coverage.

So my suggestion is <UseDataCollector/> and post the FCC output.

tonyhallett commented 3 years ago

There were no run settings used in the tests that you posted. Should there have been ?

madmoravian commented 3 years ago

Using this is my output:

--beginning of output --

Fine Code Coverage : ================================== START ==================================

Fine Code Coverage : Run Coverage Tool (disguisedname.IntegrationTests)

Fine Code Coverage : Coverlet Collector Run (disguisedname.IntegrationTests) Arguments "C:\Users\wzrubek\Documents\DisguisedProjects\disguisedname\disguisedname.IntegrationTests\bin\Debug\net5.0\fine-code-coverage\build-output\GatewayDataServices.IntegrationTests.dll" --blame --nologo --diag "C:\Users\wzrubek\Documents\DisguisedProjects\disguisedname\disguisedname.IntegrationTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output/diagnostics.log" --settings "C:\Users\wzrubek\Documents\DisguisedProjects\disguisedname\disguisedname.IntegrationTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output/FCC.runsettings" --results-directory "C:\Users\wzrubek\Documents\DisguisedProjects\disguisedname\disguisedname.IntegrationTests\bin\Debug\net5.0\fine-code-coverage\coverage-tool-output"

-- end of output

This is about an hour after starting the run.

tonyhallett commented 3 years ago

Did you provide your own runsettings ? Is the generated FCC.runsettings as expected ?

tonyhallett commented 3 years ago

If you decide to debug the vsix the entry point is https://github.com/FortuneN/FineCodeCoverage/blob/97ae2ba31e6799c332bcfa26b6db5fabe3ad2af3/FineCodeCoverage/Impl/TestContainerDiscovery/TestContainerDiscoverer.cs#L118

and this is where fcc starts the coverage for the data collector https://github.com/FortuneN/FineCodeCoverage/blob/97ae2ba31e6799c332bcfa26b6db5fabe3ad2af3/FineCodeCoverage/Core/Coverlet/DataCollector/CoverletDataCollectorUtil.cs#L189

madmoravian commented 3 years ago

RunSettingsOnly is set to true. I'm not certain how I would provide my own runsettings? I see no FCC.runsettings file anywhere in the directory structure.

tonyhallett commented 3 years ago

I'm not certain how I would provide my own runsettings?

https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019

I see no FCC.runsettings file anywhere in the directory structure.

Well that is strange. I did not use Path.Join before. In case that is an issue for you here is another vsix that does and will log when it is saving the generated run settings which is necessary for configuring coverlet when using the collector.

tonyhallett commented 3 years ago

Are you using NUnit ExplicitAttribute ?

tonyhallett commented 2 years ago

@madmoravian Revisiting this issue.

You did not respond to

Just so you are aware FCC copies your test project output to a sub directory. Perhaps your paths are incorrect.

Does this affect Fakes ?

Ideally you can create a minimal project that demonstrates the issue your are encountering.

tonyhallett commented 2 years ago

If you could run coverlet directly with the same arguments provided by FCC ( see the FCC Output Window Pane ) but with the real test dll and not the one copied by FCC.