Issue with Microsoft.CodeCoverage.Analysis

[XJPan-Jeff]

Hello,

I am measuring the code coverage of a huge desktop app which is written mainly in C++.

[Problem Conext] The basic workflow -

1. Instrument DLLs via vsinstr.exe
2. Start VSPerfMon via VSPerfCmd
3. Run test cases
4. Stop VSPerfMon and un-instrument DLLs
5. Convert the binary coverage file to XML, via a home-grown program
6. Generate coverage report via tool from https://github.com/danielpalme/ReportGenerator

Probem is with step 5 above. We wrote this program following https://github.com/danielpalme/ReportGenerator/wiki/Visual-Studio-Coverage-Tools a few years ago. We referenced Microsoft.VisualStudio.Coverage.Analysis.dll, Microsoft.VisualStudio.Coverage.Interop.dll and Microsoft.VisualStudio.Coverage.Symbols.dll in version 14.0.2.

Problem 1: The program runs out of memory if it is to convert a coverage file > 1.2 GB. As my checking, those DLLs are 32bit based, so I doubt they are limited in memory comsumption.

Problem #2: Recently I am updating the program with new DLLs in v17.0, but with slight different names: [Uploading CodeCoverage.zip…](), Microsoft.CodeCoverage.IO.dll, Microsoft.Internal.VisualStudio.Interop.dll and Microsoft.VisualStudio.Coverage.Symbols.dll. There are 2 problems - 2.1 The program can run but quit with alert that it can't load the correct DLL for symbol parsing 2.2 So I put in the Microsoft.VisualStudio.Coverage.Symbols.dll v14 again. The program runs but never returns, it hangs.

Can you please help check the issues above? I am hoping to continue the workflow and tools, with the capability to convert large coverage binaries. Many thanks!

Thanks, Jeff

[fhnaseer]

Microsoft.CodeCoverage.IO is obsolete and is no longer maintained. However Microsoft.CodeCoverage.Analaysis is shipped with latest VS and is also available here. You can use AnalysisCoverageFileUtility to merge (convert) report to xml format. Please note that these are internal tools and you are using it at your own risk. We can break public APIs in future releases.

I would also like you to explore Microsoft.CodeCoverage.Console tool to instrument and calculate code coverage for C++ applications.

[XJPan-Jeff]

Many thanks @fhnaseer !

1. With the AS-IS process, I tried AnalysisCoverageFileUtility in the new Microsoft.CodeCoverage.Analaysis but it reported that the coverage file is not in valid format. It looks the dll does not recognize the output from VSPerfMon.

2. Then I tried Microsoft.CodeCoverage.Console tool with server mode, but I am blocked at the instrument step. I am sure I have built my application with the linker option /PROFILE, then I can instrument small DLL like in 10+MB size but I can't instrument bigger DLLs with error message "Link with the /PROFILE linker swith, ensure FASTLINK is disabled, and that the PDBis writable". The tool looks quite dependent on the DLL size.

The console tool also does not recognize the coverage file produced from AS-IS process.

So, neither resolved my problems... Any further insight?

[fhnaseer]

1. Can you please share your coverage file?

2. Can you please enable logs for the instrumentation command and share these?

Console tool cannot understand VSPerfMon coverage format but it should be able to understand report generated using server mode.

[XJPan-Jeff]

1. Attached please find one sample coverage file from VSPerfMon. custom_coverage.zip

2. Attached please find the log of instrumentation command - I tried to instrument a couple of DLLs with option -ll Verbose. The log looks quite trival. instrument.log

[fhnaseer]

1. Can you please confirm that original dlls and pdbs are present in the same folder as of coverage file? ReadCoverageReport api also takes executable paths and symbols path as input.

2. Can you please confirm that following things for instrumentation. 2.1. /PROFILE is set 2.2. DebugFull full debug info 2.3. Make sure also debug information for all dependent static libraries are available 2.4. pdb is available when instrument command is called

[XJPan-Jeff]

1. For the first question -
   • Here is screenshot of the folder - the dll, pdb and coverage file are present

• Here is the screenshot of the error

2. For the 2nd query - 2.1. /PROFILE is set --- Yes 2.2. DebugFull full debug info ----NO, but I assume /PROFILE implies Debug:full 2.3. Make sure also debug information for all dependent static libraries are available --- May be not, because we reference vtksys and tbb, which have no debug info. But vsinstr can instrument well 2.4. pdb is available when instrument command is called --- YES

[fhnaseer]

1. Folder looks fine. Though you are using an older version of Microsoft.CodeCoverage.Analysis. Please use 17.11 or later.

   var utility = new AnalysisCoverageFileUtility(coverageFileConfiguration);
   utility.ReadCoverageReportAsync(path, cancellationToken);

   2.2. Try to set and see it problem persists. 2.3. Vsinstr and CodeCoverage.Console use different instrumentation mechanism. If problem is not solved, then please create an issue here https://developercommunity.visualstudio.com.

[XJPan-Jeff]

Thanks @fhnaseer !

I have created an issue on CodeCoverage.Console instrumentation https://developercommunity.visualstudio.com/t/Cant-instrument-DLLs-using-MicrosoftCo/10645886.

W.r.t the conversion, I download the Analysis lib v17.11 and copy your sample code into my program, but the build fails because it references Microsoft.CodeCoverage.Core, which I can't find it anywhere.

May you please publish a full sample code and necessary DLLs in a channel? It is hard to know the how-to to utilize the new DLLs with no sample or API helps.

Thanks, Jeff

[fhnaseer]

Nuget should add Microsoft.CodeCoverage.Core dependency as well when you add CC.Analysis package from the feed. You can download it from here.

These APIs change with time to time and there are breaking changes, so we do not have sample usage.

[XJPan-Jeff]

Tried the new package, the program failed with exception -

Here is my code snippet -

`
Microsoft.CodeCoverage.Core.CoverageFileConfiguration coverageFileConfiguration = new CoverageFileConfiguration() { ReadModules = true, ReadSkippedFunctions = true, ReadSnapshotsData = true, ReadSkippedModules = true, GenerateCoverageBufferFiles = true, FixCoverageBuffersMismatch = true };

        var utility = new AnalysisCoverageFileUtility(coverageFileConfiguration);
        var task = utility.ReadCoverageReportAsync(coverageFilePath, new CancellationToken(false));`

`

[fhnaseer]

Can you share coverage file + dlls + pdbs for conversion? And original dlls for instrumentation? You can share it on the dev community ticket privately.

[XJPan-Jeff]

I am consulting the security team in my organization about the compliance to share the PDBs. Will let you know soon.

BTW, according to https://stackoverflow.com/questions/24451827/programmatically-convert-coverage-file-to-coveragexml?newreg=4171eec4cb6649bb96b9ac6989963cf1, there seems x64 build of analysis and interops DLL in lower version like v14.

Is there any chance to get the x64 DLLs?

[fhnaseer]

The API and package mentioned here is very old. CC.Analysis 17.11.0 package contains both x86 and x64 dlls. So, it should be able to open large files without giving memory exception. I tried to open your coverage file. It seemed to be a correct coverage file, but it was not able to find original dlls and pdbs and hence it was giving exception.

Please ensure that Microsoft.CodeCoverage.Analysis.dll, Microsoft.CodeCoverage.Symbols.dll and Microsoft.CodeCoverage.Core.dll is present in your conversion utility exe folder.

I am consulting the security team in my organization about the compliance to share the PDBs. Will let you know soon.

BTW, according to https://stackoverflow.com/questions/24451827/programmatically-convert-coverage-file-to-coveragexml?newreg=4171eec4cb6649bb96b9ac6989963cf1, there seems x64 build of analysis and interops DLL in lower version like v14.

Is there any chance to get the x64 DLLs?

[XJPan-Jeff]

Add the dll / pdb and coverage binary to https://developercommunity.visualstudio.com/t/Cant-instrument-DLLs-using-MicrosoftCo/10645886?viewtype=all&moreInfo=true.

[fhnaseer]

Closing this ticket. Follow here => https://developercommunity.visualstudio.com/t/Cant-instrument-DLLs-using-MicrosoftCo/10645886#T-ND10661701