No coverage for async methods with RunMsCodeCoverage

daryl0313 commented 2 years ago

Installed product versions

Visual Studio: 2022 Professional
This extension: 1.1.173

Description

Getting no coverage results for async methods with RunMsCodeCoverage set to Yes.

Steps to recreate

I created a simple solution that reproduces the issue for Framework and dotnet core project versions.

Clone this repo: https://github.com/daryl0313/FCCAsyncTestIssue
Verify that RunMsCodeCoverage is set to Yes
Run all tests

Current behavior

Only seeing coverage for synchronous methods in the coverage report and in the gutter in the editor. Seeing 3 covered lines for each test project in the report.

Expected behavior

See coverage for all methods in the coverage report and in the gutter in the editor. Should see 6 covered lines for each test project in the report.

Side Notes

Sounds like it could be related to this: https://github.com/coverlet-coverage/coverlet/issues/794#issuecomment-612067552
- I can't see the command that FCC is using to run the tests and generate the coverage xml file in the logs.
- I don't see any option to not exclude that attribute. Even with ExcludeByAttribute cleared out, I get the same result.

Coverage Log

FCC Output

Fine Code Coverage : Ms code coverage

Fine Code Coverage : ReportGenerator Run Arguments [reporttype:Cobertura] 
"-targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\a0534286-1257-4f3c-ad4f-732653dc99c9\derek.batenhorst_DEV183_2022-06-15.11_36_17.cobertura.xml"
"-reporttypes:Cobertura"

Fine Code Coverage : ReportGenerator Run [reporttype:Cobertura]
2022-06-15T11:36:23: Arguments
2022-06-15T11:36:23:  -targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output
2022-06-15T11:36:23:  -reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\a0534286-1257-4f3c-ad4f-732653dc99c9\derek.batenhorst_DEV183_2022-06-15.11_36_17.cobertura.xml
2022-06-15T11:36:23:  -reporttypes:Cobertura
2022-06-15T11:36:24: Writing report file 'C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml'
2022-06-15T11:36:24: Report generation took 0.2 seconds

Fine Code Coverage : ReportGenerator Run Arguments [reporttype:HtmlInline_AzurePipelines] 
"-targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml"
"-plugins:C:\USERS\DEREK.BATENHORST\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_AF1CC7F5\EXTENSIONS\ZNCDRFMP.P1L\ReportGeneratorPlugins.dll"
"-reporttypes:FccLight"
"riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30"
"riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15"
"riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200"

Fine Code Coverage : ReportGenerator Run [reporttype:HtmlInline_AzurePipelines]
2022-06-15T11:36:24: Arguments
2022-06-15T11:36:24:  -targetdir:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output
2022-06-15T11:36:24:  -reports:C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\Cobertura.xml
2022-06-15T11:36:24:  -plugins:C:\USERS\DEREK.BATENHORST\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\17.0_AF1CC7F5\EXTENSIONS\ZNCDRFMP.P1L\ReportGeneratorPlugins.dll
2022-06-15T11:36:24:  -reporttypes:FccLight
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15
2022-06-15T11:36:24:  riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200
2022-06-15T11:36:25: Writing report file 'C:\play\FCCAsyncTestIssue\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output\index.html'
2022-06-15T11:36:25: Report generation took 0.2 seconds

Fine Code Coverage : Processing cobertura

Fine Code Coverage : Processing report

Fine Code Coverage : ================================== DONE ==================================

tonyhallett commented 2 years ago

Sounds like it could be related to this: https://github.com/coverlet-coverage/coverlet/issues/794#issuecomment-612067552

No. As the coverage is not being collected by coverlet.

I believe that the problem is due to the version of ReportGenerator that FCC is currently using.

I am working on a feature branch https://github.com/tonyhallett/FineCodeCoverage/tree/webview2-reportgenerator-api that uses ReportGenerator 5.1.4 as a nuget reference compared to current that uses reportgenerator.exe 4.7.1. The feature branch does not have the issue.

If you look in ...FCCAsyncTestIssue-master\FCCAsyncTestIssue-master\CoreLibrary.Tests\bin\Debug\net6.0\fine-code-coverage\coverage-tool-output

there is a Cobertura.xml file that ReportGenerator has generated from the ms cobertura.xml. You can see that the ms one is correct and the ReportGenerator one is not.

This will be rectified once I complete and push the feature.

If you need the fix now I can send you a vsix of the feature build but note the following:

It relies upon WebView2. That you have the Evergreen Runtime installed https://docs.microsoft.com/en-us/microsoft-edge/webview2/concepts/distribution#the-evergreen-runtime-distribution-mode ( When I finished the feature FCC will install it for you if necessary )

The report is not currently reflecting visual studio theming / environment fonts.

The report does not currently have sticky tables.

JayOwl commented 2 years ago

I'm not sure it's related to this issue but I got my async methods to be covered by changing the setting Tools > Options > Fine Code Coverage > AdjacentBuildOutput to true. Thanks Jeremy Thompson.

viceice commented 2 years ago

@JayOwl Are you using coverlet or Microsoft.CodeCoverage ? I think that workaround doesn't work for Microsoft.CodeCoverage

@tonyhallett Any chance to get a release with new feature? Or at least a prerelease to test 🙃

tonyhallett commented 2 years ago

@viceice

I think that workaround doesn't work for Microsoft.CodeCoverage Agreed

The new feature now has the necessary code for the WebView2 runtime so what remains is the styling of the react report. I should be able to complete this in the next two weeks.

viceice commented 2 years ago

Sorry for noise. 🤗 Any news? It's pretty anoying to not see async function coverage. 😕

tonyhallett commented 2 years ago

Has been fixed https://github.com/FortuneN/FineCodeCoverage/commit/1f040cb5de9cad8942a42f1ac6ac1f4eb79681b6 but..........

@FortuneN Your publisher access token has expired https://github.com/FortuneN/FineCodeCoverage/runs/8041220586?check_suite_focus=true

The new release is not my new feature of WebView2 + React report. This I will finish when I return from holiday.

To have ms coverage async methods working we cannot just update to the latest report generator version that fixed https://github.com/danielpalme/ReportGenerator/issues/467 as an earlier commit removed ie and hence WebBrowser control support - https://github.com/danielpalme/ReportGenerator/commit/f801790d80ebf76030476217848e98d05c8075b7

So I have forked the repo at 4.7.1 and added the single line of code that fixes async. This code is https://github.com/FortuneN/FineCodeCoverage/blob/master/Shared%20Files/ZippedTools/reportGenerator.4.7.2.zip and https://github.com/FortuneN/FineCodeCoverage/blob/master/Shared%20Files/ZippedTools/ReportGenerator.Core.dll

So please update your access token !

tonyhallett commented 2 years ago

Although the vsix marketplace release github action did not occur, the release action did.

https://github.com/FortuneN/FineCodeCoverage/releases/tag/v1.1.181

So you can use https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.181/FineCodeCoverage2022.vsix https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.181/FineCodeCoverage.vsix

viceice commented 2 years ago

will test it on Monday ❤️

FortuneN commented 2 years ago

Done. https://github.com/FortuneN/FineCodeCoverage/runs/8052519049?check_suite_focus=true

viceice commented 2 years ago

Publish was skipped, so still missing on store

tonyhallett commented 2 years ago

@FortuneN

https://github.com/FortuneN/FineCodeCoverage/runs/8188031339?check_suite_focus=true

add to marketplace

VSSDK: error VsixPub0031 : An error occurred while attempting to authenticate: VS30063: You are not authorized to access https://marketplace.visualstudio.com./

Once rectified the workflow cannot be re-run but needs to be triggered. Let me know when ready. Thanks

tonyhallett commented 2 years ago

https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage.vsix https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage2022.vsix

This release fixes generic classes with ms code coverage.

kakins commented 2 years ago

https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage.vsix https://github.com/FortuneN/FineCodeCoverage/releases/download/v1.1.184/FineCodeCoverage2022.vsix

This release fixes generic classes with ms code coverage.

If I'm not mistaken then, this release also includes fixing async methods, correct? It appears to be working for me now

tonyhallett commented 2 years ago

@kakins correct

viceice commented 2 years ago

@FortuneN It's still not publishing to the store 😕

FortuneN commented 2 years ago

I've updated the token. The only issue on my run is the version which i'm sure is a nothing-burger; right @tonyhallett ? https://github.com/FortuneN/FineCodeCoverage/actions/runs/3023299512/jobs/4904846997

tonyhallett commented 2 years ago

@FortuneN You cannot use workflow_dispatch. It needs to be triggered by pull request merge. I do one shortly.

At a later date can look at the actions further.

tonyhallett commented 2 years ago

@FortuneN Still getting the access token issue https://github.com/FortuneN/FineCodeCoverage/actions/runs/3045484608/jobs/4907101471

Error: Error: Command failed: VsixPublisher.exe publish -payload D:\a\FineCodeCoverage\FineCodeCoverage\FineCodeCoverage\bin\Release\FineCodeCoverage.vsix -publishManifest D:\a\FineCodeCoverage\FineCodeCoverage\vs-market-place-manifest.json -personalAccessToken *** VSSDK: error VsixPub0031 : An error occurred while attempting to authenticate: VS30063: You are not authorized to access https://marketplace.visualstudio.com./

FortuneN commented 2 years ago

Publish seems to be resolved.

Not sure about that last 'comment' bit

https://github.com/FortuneN/FineCodeCoverage/actions/runs/3077073438/jobs/4971676090

tonyhallett commented 1 year ago

Thanks. I will kick off the process again. At a later date the workflows can be revisited to support manual triggers.

On Tue, 13 Sept 2022 at 12:04, Fortune Ngwenya @.***> wrote:

[image: image] https://user-images.githubusercontent.com/5636175/189885287-e5e38417-6106-4831-b8e7-14d3a60ac213.png

I updated the token

— Reply to this email directly, view it on GitHub https://github.com/FortuneN/FineCodeCoverage/issues/270#issuecomment-1245250766, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACWFCRUZEHNEEFNRZZ5JE3LV6BNUNANCNFSM5Y36PSMQ . You are receiving this because you were mentioned.Message ID: @.***>

FortuneN / FineCodeCoverage