How to collect coverage of third party libraries?

fuzzah commented 1 year ago

Hello! I am testing some app utilizing Newtonsoft.Json library from NuGet. I have sources for both app and the library (same version as installed from NuGet, but simply cloned from github). How can I point AltCover to location of Newtonsoft.Json sources?

Here's what I use to instrument my assemblies:

altcover -y ~/Newtonsoft.Json/ -i ~/myapp_publish/ --inplace

myapp.dll gets instrumented, but Newtonsoft.Json.dll is left intact and coverage reports contain no references to this third party library (I need both in one report)

SteveGilham commented 1 year ago

With the Newtonsoft source, you could simply build the libraries as part of your system, so removing the third-party nature of the code.

The process comes in two stages - first, instrumentation which determines from debug information which instructions in the assembly to trace, and what line/column/file that corresponds to. Then, having executed the instrumented code, matching the paths in the coverage report to local files for e.g. ReportGenerator (or the AltCover Visualizer).

As it stands, the NuGet package Newtonsoft.Json doesn't include debug symbols, and that's where the process then fails to consider the assemblies of interest.

So, first, adding the symbols -- these are found in a separate package e.g. https://www.nuget.org/api/v2/symbolpackage/Newtonsoft.Json/13.0.1 for the most recent non-preview release. Downloading that and unpacking to co-locate the symbols with the assembly for the corresponding platform would pick up the symbols automatically; alternatively, for a .snupkg unzipped to an arbitrary location, the -y directory needed would be <root of unzipped .snupkg>/lib/<net version>.

Having done that, you get a coverage file containing (in OpenCover format) something like

 <Modules>
    <Module hash="E3-67-3D-05-D4-6F-29-E6-82-41-D4-53-6B-DD-F1-8C-DD-0A-91-3D">
      <Summary numSequencePoints="17127" visitedSequencePoints="0" numBranchPoints="13821" visitedBranchPoints="0" sequenceCoverage="0" branchCoverage="0" maxCyclomaticComplexity="85" minCyclomaticComplexity="1" visitedClasses="0" numClasses="393" visitedMethods="0" numMethods="3086" minCrapScore="0" maxCrapScore="0" />
      <ModulePath>[path to]\Newtonsoft.Json.dll</ModulePath>
      <ModuleTime>2021-03-17T20:03:36Z</ModuleTime>
      <ModuleName>Newtonsoft.Json</ModuleName>
      <Files>
        <File uid="1" fullPath="/_/Src/Newtonsoft.Json/Utilities/NullableAttributes.cs" />
        <File uid="2" fullPath="/_/Src/Newtonsoft.Json/DefaultJsonNameTable.cs" />
        <File uid="3" fullPath="/_/Src/Newtonsoft.Json/JsonArrayAttribute.cs" />

At some point after generating the initial report, either before or after doing the test run, you then need the manual step to change /_/Src/Newtonsoft.Json to (in your case) ~/Newtonsoft.Json globally in the file to pick up the local source paths. Or, in this case, with the report containing sourcelink file paths, the reporting tool you use may be able to locate the source file through that mechanism.

If you add --sourcelink to the AltCover command line, the report will be generated with the corresponding URLs, like

      <Files>
        <File uid="1" fullPath="https://raw.githubusercontent.com/JamesNK/Newtonsoft.Json/ae9fe44e1323e91bcbd185ca1a14099fba7c021f/Src/Newtonsoft.Json/Utilities/NullableAttributes.cs" />
        <File uid="2" fullPath="https://raw.githubusercontent.com/JamesNK/Newtonsoft.Json/ae9fe44e1323e91bcbd185ca1a14099fba7c021f/Src/Newtonsoft.Json/DefaultJsonNameTable.cs" />
        <File uid="3" fullPath="https://raw.githubusercontent.com/JamesNK/Newtonsoft.Json/ae9fe44e1323e91bcbd185ca1a14099fba7c021f/Src/Newtonsoft.Json/JsonArrayAttribute.cs" />

and the Visualizer, at least, will follow those URLs when displaying coverage (it doesn't do the sourcelink lookup via the .pdb file itself, but relies on it having been done at report generation time).

fuzzah commented 1 year ago

Thank you! Was foolish of me to not know about the whole snupkg thing :) Putting pdb next to dll worked like a charm.

Some notes

which pdb to use

snupkg will contain pdb files for multiple .NET versions, if not sure what's needed, one may use strings + grep on dll like so:

strings -el Newtonsoft.Json.dll | grep -i standard

In my case this command returned: Json.NET .NET Standard 2.0 That means we need lib/netstandard2.0/Newtonsoft.Json.pdb from snupkg.

how to replace paths in LCOV info files

When using lcov format, we can easily fix paths by using sed, e.g.,

sed -i cov.info -e 's|/_/Src|/home/user/Newtonsoft.Json/Src|g'

SteveGilham / altcover