coverlet-coverage / coverlet

Cross platform code coverage for .NET
MIT License
2.99k stars 386 forks source link

Coverage results show 0 line hits #1487

Closed Inirit closed 1 year ago

Inirit commented 1 year ago

I'm running into a situation where the coverage results show 0 line hits. I know for sure this cannot be the case as my tests successfully execute, so there must be something else going on.

I'm aware of the Known Issues page which describes exactly what I'm observing: 0 line hits and a message in the diagnostic logs saying "Hits file:'X' not found for module: 'Y'". However, the solution says to use collectors which I already am. The command used to invoke the tests looks something like this:

dotnet test "MyTestAssembly.dll" --collect:"XPlat Code Coverage" --logger:"console;verbosity=detailed" --results-directory:"results" --TestAdapterPath:"PathToCoverletNuGetPackageFiles/build/netstandard1.0" --settings:"MyRunsettings.runsettings"

(A few things to note: I am working entirely outside of MSBuild and I'm invoking "dotnet test" against an already-built dll)

In the diagnostic logs for the dotnet testhost, these lines (which correspond to the only test that exists in the test project) seem possibly related but I don't know for sure:

TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.463, 2349562112263700, testhost.dll, TestExecutionRecorder.RecordStart: Starting test: Example.Library.Test.Tests.JSON_Serialize. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.496, 2349562145105500, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseStart: Invoking callback 1/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager., took 24 ms. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.496, 2349562145510300, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseStart: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager., took 0 ms. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.508, 2349562156813300, testhost.dll, TestExecutionRecorder.RecordEnd: test: Example.Library.Test.Tests.JSON_Serialize execution completed. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.512, 2349562160923600, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseEnd: Invoking callback 1/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager., took 3 ms. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.512, 2349562161371500, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseEnd: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager., took 0 ms. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.513, 2349562162444700, testhost.dll, TestExecutionRecorder.RecordResult: Received result for test: Example.Library.Test.Tests.JSON_Serialize. TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.515, 2349562164079300, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseEnd: Invoking callback 1/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyOutOfProcDataCollectionManager., took 1 ms. TpTrace Warning: 0 : 121, 15, 2023/06/05, 19:14:48.516, 2349562165232700, testhost.dll, No DataCollection Data set for the test case 670de377-4d5a-7ef3-2712-b4e74059d9ac TpTrace Verbose: 0 : 121, 15, 2023/06/05, 19:14:48.516, 2349562165306900, testhost.dll, MulticastDelegateUtilities.SafeInvoke: TestCaseEventsHandler.RaiseTestCaseEnd: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.InProcDataCollectionExtensionManager., took 1 ms. TpTrace Warning: 0 : 121, 15, 2023/06/05, 19:14:48.519, 2349562168165600, testhost.dll, TestRunCache: InProgressTests is null TpTrace Warning: 0 : 121, 15, 2023/06/05, 19:14:48.519, 2349562168287300, testhost.dll, TestRunCache: No test found corresponding to testResult 'JSON_Serialize' in inProgress list.

The logs from the data collector that cover the time frame of the above logs:

TpTrace Verbose: 0 : 100, 1, 2023/06/05, 19:14:47.732, 2349561381250100, datacollector.dll, MulticastDelegateUtilities.SafeInvoke: DataCollectionEvents.SessionStart: Invoking callback 1/1 for Coverlet.Collector.DataCollection.CoverletCoverageCollector.OnSessionStart, took 378 ms. TpTrace Information: 0 : 100, 1, 2023/06/05, 19:14:47.737, 2349561385759800, datacollector.dll, DataCollectionRequestHandler.ProcessRequests : DataCollection started. TpTrace Information: 0 : 100, 1, 2023/06/05, 19:14:47.751, 2349561399824000, datacollector.dll, DataCollectionRequestHandler.ProcessRequests: Datacollector received message: (DataCollection.TestHostLaunched) -> {"MessageType":"DataCollection.TestHostLaunched","Payload":{"ProcessId":121}} TpTrace Verbose: 0 : 100, 1, 2023/06/05, 19:14:47.752, 2349561400895300, datacollector.dll, MulticastDelegateUtilities.SafeInvoke: DataCollectionEvents.TestHostLaunched: Invoking callbacks was skipped because there are no subscribers. TpTrace Information: 0 : 100, 1, 2023/06/05, 19:14:48.781, 2349562429845800, datacollector.dll, DataCollectionRequestHandler.ProcessRequests: Datacollector received message: (DataCollection.AfterTestRunEnd) -> {"MessageType":"DataCollection.AfterTestRunEnd","Payload":false} TpTrace Verbose: 0 : 100, 1, 2023/06/05, 19:14:48.783, 2349562432076900, datacollector.dll, [coverlet]CoverletCoverageDataCollector: SessionEnd received TpTrace Verbose: 0 : 100, 1, 2023/06/05, 19:14:48.796, 2349562445333100, datacollector.dll, [coverlet]Hits file:'/tmp/Example.Library_282ba6c0-64fa-48e2-b12b-cd08f9930a49' not found for module: 'Example.Library' TpTrace Information: 0 : 100, 1, 2023/06/05, 19:14:48.831, 2349562480402600, datacollector.dll, [coverlet]CoverletCoverageDataCollector: Saved coverage report to path: '/tmp/7e61f5dc-c1ab-4d54-acda-e35557c616d4/coverage.cobertura.xml'

Beyond this I'm not sure what else to look at to diagnose the problem.

Inirit commented 1 year ago

Well I figured out what the problem was and it was not necessarily related to coverlet. It would take a lot to explain the entire situation, but the sort of short version is that our build output directory is composed entirely of symlinks but with a few files hard copied so that Coverlet could instrument them without polluting the original files that could be referenced by other build targets. However, the dotnet runtime was still resolving paths of dlls in this original physical location that the symlinks (replaced by hard copies) were pointing to due to the deps.json file associated with the test assembly still being a symlink. On Linux, the dotnet runtime resolves the realpath of this file (which fully resolves symlinks) and looks for dlls adjacent to that resolved path (I'm about 80% sure about this, would need to look through the runtime source code again). Coverlet would correctly instrument the hard copied files, but dotnet would load the untouched original files and therefore run tests without instrumentation and therefore without generating hit files.

The fix was to hard copy the deps.json file into the build output directory.