search

sdavis3 / code-coverage-widgets

A dashboard widget for Azure DevOps or Team Foundation Server (TFS) to display information about unit test code coverage.
MIT License
16 stars 10 forks source link

Coverage Result Discrepancy #34

Closed JimmyMac6996 closed 3 years ago

JimmyMac6996 commented 4 years ago

More a question than an issue (possibly) we are running coverlet and producing our coverage report in the Cobertura format for visibility to our code coverage in each run in the ADO code coverage tab. Our PM wanted visibility on the dashboard so I gave this a try. In the pipeline it is showing 25.5% (1131 of 4431) for the line coverage, but in the widget it is showing 30.54 Percent (Lines) - is there a know reason for the difference? Thanks

sdavis3 commented 4 years ago

Sorry for the delay...just saw this issue. I'm not 100% familiar with Coverlet, but have you tried selecting "Line" instead of "Lines"? Do you get a different result? Does the 4431 reference anything specific that you are aware of? Is it the number of lines of code that also includes comments maybe?

I don't do any calculation, per say, but rather read the output coming from the API.

Can you attach some screenshots so I can correlate where you are seeing these exact numbers please?

sdavis3 commented 4 years ago

@JimmyMac6996 Following up on this to see if you're able to answer the questions above and post a screenshot or two. Thank you.

bmarinov commented 4 years ago

@sdavis3 I encountered the same problem. Code coverage seemed a bit off so I checked the generated reports. The summary of the cobertura.xml report shows the same value as the widget. But the code coverage summary in the build is wrong / off / calculated separately.

Screenshots of an example project:

cobertura.xml

<coverage line-rate="0.39477726574500765" branch-rate="0.29411764705882354" lines-covered="771" lines-valid="1953" branches-covered="100" branches-valid="340" complexity="NaN" version="0" timestamp="*******">

The widget then displays the information from the cobertura summary. branch-rate="0.29411764705882354" image

Summary in the build (calculated correctly according to the stats in the project breakdown of the HTML report): image

Branch coverage here is at 26.6%, which does not correspond to the cobertura.xml report.

The question is, why is there a discrepancy between the cobertura summary and the raw data.

Maybe https://github.com/danielpalme/ReportGenerator is the culprit or we are using it wrong. Gonna investigate more.

sdavis3 commented 3 years ago

@bmarinov Were you ever able to get any resolution to your question?

bmarinov commented 3 years ago

@bmarinov Were you ever able to get any resolution to your question?

If I remember correctly, the ReportGenerator version on Azure DevOps was very outdated. This could be seen in one of the pipelien tasks on github. I ended up installing it as a global tool and using the newest version that way. @sdavis3 hope this helps :)