CollectFromChildProcesses supported?

[candritzky]

With the Microsoft CodeCoverage collector it is possible to collect code coverage from child processes via the following setting in the .runsettings file:

<CollectFromChildProcesses>True</CollectFromChildProcesses>

It this also possible with coverlet.collector?

[MarcoRossignoli]

Interesting,

I think it should work by design.

Coverlet instruments dlls and on process exists it persists hits/probes. So if your child process(started from tests) loads instrumented dlls, on process exits, hits will be saved and accounted as expected.

[candritzky]

@MarcoRossignoli Thanks, this is good news.

Maybe it doesn't work because the child process started (and terminated) by our test resides in a different bin directory than the test assemblies. They are part of the same repo, same solution and same build, but they are built to and started from a different bin directory. The structure is like that:

MyProgram.sln

MyProgram\MyProgram.csproj
MyProgram\bin\Debug\net5.0\MyProgram.exe
MyProgram\bin\Debug\net5.0\MyProgram.dll
MyProgram\bin\Debug\net5.0\MyProgram.pdb

MyProgram.E2ETests\MyProgram.E2ETests.csproj
MyProgram.E2ETests\bin\Debug\net5.0\MyProgram.E2ETests.dll

The test code in MyProgram.E2ETests.dll traverses up to the source code root dir and launches MyProgram.exe from the sibling MyProgram\bin\Debug\net5.0 directory.

Could that be the problem? How does Coverlet decide which DLLs to instrument?

[candritzky]

Maybe I have to use the IncludeDirectory setting and point it to the MyProgram source (or rather bin?) directory.

[MarcoRossignoli]

Yep with IncludeDirectory should work, to understand if you're instrumenting correct libraries use the logs https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md#collectors-integration --diag

[candritzky]

I finally got the IncludeDirectory syntax right and I can see the child process assembly being instrumented:

TpTrace Verbose: 0 : 48704, 1, 2021/11/23, 19:33:28.083, 420350525559, datacollector.dll, [coverlet]Instrumented module: 'D:\Git\MyRepo\MyProgram\bin\Debug\net5.0\win-x64\MyProgram.dll'

The module then also shows up in the generated coverage.cobertura.xml file as <package name="MyProgram" line-rate="0" branch-rate="0" complexity="94">....

But all the line-rates and hits for this module are still 0.

@MarcoRossignoli Do you have a working example for child process code coverage?

[MarcoRossignoli]

Unfortunately nope, I need to repro one, I'll do asap to verify if my assumption was correct or if there are other issue with this scenario.

[candritzky]

@MarcoRossignoli Thanks for trying a repro.

Just for reference, here's my .runsettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <TargetPlatform>x64</TargetPlatform>
  </RunConfiguration>

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat Code Coverage">
        <Configuration>
          <Format>Cobertura</Format>
          <IncludeDirectory>D:\Git\MyRepo\MyProgram\bin\Debug\net5.0\win-x64</IncludeDirectory>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

  <MSTest>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>True</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>
</RunSettings>

And here's how I execute the tests from the command line:

cd /d D:\Git\MyRepo
dotnet test MyProgram.E2ETests --settings MyProgram.runsettings --diag:log.txt

[MarcoRossignoli]

Got it maybe, you need to specify in command line --collect:"XPlat Code Coverage" coverlet relies on an InProcessDataCollector that is inserted only if specified on command line (one thing that we need to improve)

[candritzky]

@MarcoRossignoli Thanks but adding --collect:"XPlat Code Coverage" in the command line didn't help either. Same result.

I assumed specifying a .runsettings file (via --settings) with a properly configured DataCollector would be equivalent.

I also tried adding a PackageReference to coverlet.collector into MyProgram.csproj (the executable, not the test project), but this didn't help either.

I think we need a working example/demo for the out-of-proc/child process code coverage scenario.

[MarcoRossignoli]

One question, how do you "shutdown" child process?coverlet collect hits on process exists...so the process needs to be closed gracefully(exit the "main"), a kill won't work.

[candritzky]

@MarcoRossignoli Thank you, good point. We're using Process.Kill to "shutdown" the child process. I'll try to send a Ctrl+C signal instead and see if this helps.

[MarcoRossignoli]

We have same problem for standalone run https://github.com/coverlet-coverage/coverlet/issues/781#issuecomment-691442083

[candritzky]

@MarcoRossignoli Thank you so much for your support. I finally managed to get it running and have code coverage numbers of my child process in the coverage.cobertura.xml file.

I had problems with sending a Ctrl+C signal to the child process. The code I used (based on AttachConsole and GenerateConsoleCtrlEvent as described here: https://stackoverflow.com/a/29274238/1273698) seems to work only on .NET Framework, but fails on .NET 5.0 (with error code 5 = access denied). So I first had to add a remote control channel that allowed me to send a signal from my test code to the child process which then gracefully (asynchronously) commits suicide via Environment.Exit(0).

[MarcoRossignoli]

So I first had to add a remote control channel that allowed me to send a signal from my test code to the child process which then gracefully (asynchronously) commits suicide via Environment.Exit(0).

Out of curiosity what did you use?Tcp, pipe, shared mem, file system, sync events?

[candritzky]

@MarcoRossignoli Since the out-of-proc server that we're testing is an ASP.NET Core WebAPI server, I just added an ApiController (ShutdownController) to implement this. But if it was something else, I most probably had built something based on TinyIpc.

[MarcoRossignoli]

Got it.

Feel free to close this one if solved!

[candritzky]

@MarcoRossignoli One (hopefully final) question: I'm still struggling with the IncludeDirectory configuration. I got it working by placing hardcoded fully qualified path names (e. g. "D:\Git\MyRepo...") into the .runsettings file. But this was just a workaround to get code coverage going. In the final solution I need to pass the additional include directories on the command line. I thought it could be achieved like this:

dotnet test ... --settings MyProgram.runsettings /p:IncludeDirectory=D:\Git\MyRepo\MyProgram1,D:\Git\MyRepo\MyProgram2

But this doesn't work. I also tried to put the value in escaped double quotes like this:

dotnet test ... --settings MyProgram.runsettings /p:IncludeDirectory=\"D:\Git\MyRepo\MyProgram1,D:\Git\MyRepo\MyProgram2\"

I also tried escaping the backslashes in the argument value, but nothing helped. What is the correct syntax?

[MarcoRossignoli]

/p:IncludeDirectory is used on msbuild integration not for collectors, you need to pass it using runsettings https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#advanced-options-supported-via-runsettings Or through command line https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#passing-runsettings-arguments-through-commandline on the fly

[candritzky]

I tried the command line version like this:

dotnet test ... --settings MyProgram.runsettings -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.IncludeDirectory=D:\Git\MyRepo\MyProgram\bin\Debug\net5.0\win-x64

But this is then passed to the IncludeDirectories parameters using escaped (or "doubled") backslash characters and then the additional modules are not found/not instrumented.

I also tried putting the value in \"...\" and also tried using forward slashes as the path separator char, but nothing did work (on Windows).

[candritzky]

@MarcoRossignoli OK, I see. This double backslash escaping is not a problem. Instrumentation of modules also works when the directory names have double backslashes as a path separator.

I'm closing this. Thanks a lot for your support!

[candritzky]

Happens again, created new issue #1338.