No test result file when specifying coverage, result file created correctly without coverage

[robrich]

Given:

some.sln
SomeTestProject
- some.csproj
- sometest.cs

Works as expected:

dotnet test -c Release --no-build --filter "Type!=Integration" --logger:"junit;LogFilePath=../test-results/unit-result.xml"

I now have a test-results/unit-result.xml file right next to my solution. I have no code coverage details, but I haven't asked for any.

Doesn't work:

dotnet test -c Release --no-build --filter "Type!=Integration" --logger:"xunit;LogFilePath=../test-results/unit-result.xml" --collect:"XPlat Code Coverage;Format=opencover,cobertura" --results-directory="test-results"

I have a test-results/*/coverage.*.xml but no test-results/unit-result.xml file. (I find it curious that --results-directory is relative to the current working directory and LogFilePath is relative to the csproj, but that's a problem for another day.)

Does work:

dotnet test -c Release --no-build --filter "Type!=Integration" --logger:"junit;LogFilePath=../test-results/unit-result.xml" --collect:"XPlat Code Coverage;Format=opencover,cobertura" --results-directory="test-results"

I have both test-results/unit-result.xml and test-results/*/coverage.*.xml. Note that in this case I'm /not/ using xUnit, I'm using jUnit instead.

[codito]

Could you share the log.* files generated with dotnet test --diag:log.txt command?

Which version of xunit logger and which OS? I will try to repro this.

[robrich]

I'll go one better. https://github.com/robrich/xunit-logger-bug-repro is a repository that nicely reproduces the error. I've seen it fail both on ubuntu-latest Azure DevOps runner and on Windows 11.

[codito]

@robrich thanks, the repro project was super helpful :+1:

Error stack trace

``` TpTrace Error: 0 : 80063, 4, 2023/12/30, 18:45:00.789, 9311276010677, vstest.console.dll, InternalTestLoggerEvents.SendTestRunComplete: Exception occurred while calling handler of type Spekt.TestLogger.Core.TestRunBuilder for TestRunCompleteEventArgs: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.MissingMethodException: Method not found: 'System.String Spekt.TestLogger.Core.TestResultInfo.get_DisplayName()'. at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.CreateTestElement(TestResultInfo result) at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.CreateCollection(IGrouping`2 resultsByType, List`1 testResultAsError) at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.<>c__DisplayClass9_0.b__2(IGrouping`2 resultsByType) at System.Linq.Enumerable.SelectIPartitionIterator`2.MoveNext() at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.CreateAssemblyElement(IGrouping`2 resultsByAssembly, LoggerConfiguration loggerConfiguration, TestRunConfiguration runConfiguration) at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.<>c__DisplayClass8_0.b__2(IGrouping`2 resultsByAssembly) at System.Linq.Enumerable.SelectIPartitionIterator`2.MoveNext() at System.Xml.Linq.XContainer.AddContentSkipNotify(Object content) at System.Xml.Linq.XElement..ctor(XName name, Object content) at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.CreateAssembliesElement(List`1 results, LoggerConfiguration loggerConfiguration, TestRunConfiguration runConfiguration) at Microsoft.VisualStudio.TestPlatform.Extension.Xunit.Xml.TestLogger.XunitXmlSerializer.Serialize(LoggerConfiguration loggerConfiguration, TestRunConfiguration runConfiguration, List`1 results, List`1 messages) at Spekt.TestLogger.Core.TestRunCompleteWorkflow.Complete(ITestRun testRun, TestRunCompleteEventArgs completeEvent) in /home/runner/work/testlogger/testlogger/src/TestLogger/Core/TestRunCompleteWorkflow.cs:line 37 at Spekt.TestLogger.Core.TestRunBuilder.b__5_3(Object _, TestRunCompleteEventArgs eventArgs) in /home/runner/work/testlogger/testlogger/src/TestLogger/Core/TestRunBuilder.cs:line 55 --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at System.Delegate.DynamicInvokeImpl(Object[] args) at System.Delegate.DynamicInvoke(Object[] args) at Microsoft.VisualStudio.TestPlatform.Utilities.MulticastDelegateUtilities.SafeInvoke(Delegate delegates, Object sender, EventArgs args, String traceDisplayName) ```

Both Junit logger and Xunit logger depend on the Spekt.Testlogger project. And both are using different versions of the testlogger. Xunit logger uses the latest version with coverage capabilities.

When the test project has both test logger references, MSBuild copies both logger's Spekt.TestLogger.dll to the OutputDirectory; however arbitrarily only one Spekt.TestLogger.dll. When Junit's copy of testlogger.dll remains, Xunit logger fails to emit a test result with above stacktrace.

Workaround: removing Junit logger reference from the test project fixes the issue. Does this unblock your scenario?

The fix will require a new release for Junit with latest testlogger version.

[robrich]

Yes, removing the reference to jUnit did solve the symptoms. Thanks also for the PR to the jUnit project to get it up-to-date too.