Open MarcoRossignoli opened 4 years ago
Does VSTest signal the end of the entire test run (like an event)? Each data collector could then do one last action.
No AFAIK...this is the biggest issue for coverlet for a lot of features like merging...we don't have a way to synch between parallel tests run. I'm in contact with vstest team to undestand if there is a solution, but I think that at the moment the only place is inside dotnet
cli out of vstest control, btw this is my speculation, we'll see.
For this issue I had a fast conversation with team member and seem that we could inject a custom logger https://github.com/microsoft/vstest-docs/blob/master/docs/report.md
Thanks @MarcoRossignoli for working on this issue ASAP. this is the biggest blocker for us, using the vstest collector.
@PureKrome why?Can you tell me how you use terminal data?
Sure, either when:
using the command line (eg. dotnet test ..) creates a file which I can't read easily. Just seeing the ascii art chart after that command is a great first and easy step into seeing some quick steps.
It's a blocker because it's slowing me down when coding. I don't want to have to push the report file up to codecov.io and then log in to that to review the report.
Ok for local dev you can use ReportGenerator with simple script https://github.com/danielpalme/ReportGenerator
This issue has strained on me for an hour. Due to https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application, we have been exploring vstest
instead of msbuild
for code coverage.
To my dismay, same as op, there is seemingly no way today to get a simple reST
(https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html) table summary output from this tool.
The REASON for this is for gitlab CI coverage integration and badges, see: https://docs.gitlab.com/ee/ci/yaml/#coverage, so for a gitlab job that runs dotnet test
with coverage ... one would regex match like this:
# opencover coverage report prints in TABLE reST format
coverage: '/^\s*\|\s+Total\s+\|\s+(\d+\.?\d+\%)\s+\|/'
I tried the suggested workaround (to use an open-source report generator https://danielpalme.github.io/ReportGenerator/usage.html), however this cannot achieve it either. The closest I found was using the TextSummary
dotnet reportgenerator -reports:/SNIP/coverage.opencover.xml -targetdir:coveragereport -reporttypes:TextSummary
, this gives us something like:
(snip)
...
2020-09-21T15:37:46: Writing report file 'coveragereport/Summary.txt'
2020-09-21T15:37:46: Report generation took 0.3 seconds
so then we can at least get that data with:
$ sed -n '/Summary/,/^$/p' coveragereport/Summary.txt
Summary
Generated on: 2020-09-21 - 3:33:00 p.m.
Parser: OpenCoverParser
Assemblies: X
Classes: XX
Files: XXX
Line coverage: ##.#%
Covered lines: XXXX
Uncovered lines: XXXX
Coverable lines: XXXX
Total lines: XXXX
and we'd have to augment our regex match
@mcallaghan-bsm I went through the same hoops - using a vstest "collector", then report generator for a text summary, and finally scrape the output for GitLab CI. AFAIK, this is the best way until coverlet collector hooks into collector logging to generate the text summary itself.
@mcallaghan-bsm and @gitfool - wow! as a workaround, that's great!!! painful ... but great!
We use an another approach with the coverlet vstest collector on GitLab. Coverlet is configured to output the results as opencover format. After the test run a script extracts the coverage from the report and prints it to console.
PERCENTAGE=$(grep -m 1 -o -P '(?<=sequenceCoverage=")(\d+.\d+|\d+)' $DIRECTORY/coverage.opencover.xml)
echo "Line coverage: ${PERCENTAGE}%"
This issue is stale because it has been open for 3 months with no activity.
Today at the end of the tests msbuil/.net tool drivers show a table with coverage result on console. For instance:
This is not supported at the moment for vstest collectors integration.
cc: @vagisha-nidhi @PureKrome