search

Issafalcon / neotest-dotnet

Neotest adapter for dotnet
MIT License
69 stars 23 forks source link

False positive test failure on .NET 7 #25

Closed X9VoiD closed 1 year ago

X9VoiD commented 1 year ago

I'm not quite sure if this is reproducible in other machines.

Reproduction

Install .NET 7 SDK

dotnet new xunit -n IssueRepro

Run IssueRepro.UnitTest1.Test1

Expected behavior

Should run just fine

Observed behavior

Test is successful but neovim-dotnet shows otherwise.

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="c2802456-58a3-4e8e-a738-6ad9622f522c" name="@nixos 2022-11-13 09:17:29" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2022-11-13T09:17:29.1522395+08:00" queuing="2022-11-13T09:17:29.1522396+08:00" start="2022-11-13T09:17:27.4667035+08:00" finish="2022-11-13T09:17:29.1647285+08:00" />
  <TestSettings name="default" id="26492e33-0d77-4486-b5ec-d343ae989a9a">
    <Deployment runDeploymentRoot="_nixos_2022-11-13_09_17_29" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="fd870a85-5624-4f3e-b430-2e7a991b5355" testId="de56690c-beec-fdd9-35b2-b08e53285b43" testName="IssueRepro.UnitTest1.Test1" computerName="nixos" duration="00:00:00.0027649" startTime="2022-11-13T09:17:28.9735291+08:00" endTime="2022-11-13T09:17:28.9735763+08:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="fd870a85-5624-4f3e-b430-2e7a991b5355" />
  </Results>
  <TestDefinitions>
    <UnitTest name="IssueRepro.UnitTest1.Test1" storage="/home/oscar/work/issuerepro/issuerepro/bin/debug/net7.0/tmp/nvim.oscar/rwvo9k/issuerepro.dll" id="de56690c-beec-fdd9-35b2-b08e53285b43">
      <Execution id="fd870a85-5624-4f3e-b430-2e7a991b5355" />
      <TestMethod codeBase="/home/oscar/Work/IssueRepro/IssueRepro/bin/Debug/net7.0/tmp/nvim.oscar/rwVo9k/IssueRepro.dll" adapterTypeName="executor://xunit/VsTestRunner2/netcoreapp" className="IssueRepro.UnitTest1" name="Test1" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="de56690c-beec-fdd9-35b2-b08e53285b43" executionId="fd870a85-5624-4f3e-b430-2e7a991b5355" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 7.0.0)
[xUnit.net 00:00:00.61]   Discovering: IssueRepro
[xUnit.net 00:00:00.65]   Discovered:  IssueRepro
[xUnit.net 00:00:00.65]   Starting:    IssueRepro
[xUnit.net 00:00:00.73]   Finished:    IssueRepro
</StdOut>
    </Output>
  </ResultSummary>
</TestRun>

image

Issafalcon commented 1 year ago

Thanks for raising this @X9VoiD - #28 should be the fix for this issue. Give it a go to make sure, but I'll close this issue. If you still find the mismatch between the output and the icon, feel free to re-open and I'll investigate some more.

X9VoiD commented 1 year ago

Can confirm it's fixed, thanks!

lawrence-laz commented 1 year ago

@Issafalcon I am having the same issue, but on dotnet 6: image

Is this related, or should I open a new issue? Not sure how neotest determines whether the test was successful, does it parse the command output?

EDIT: I see there is a results file, here's mine in this case:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="fa9f6046-7344-47b9-8c59-81c110e2cb96" name="@llaz-laptop 2022-12-31 14:58:32" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2022-12-31T14:58:32.9197387+02:00" queuing="2022-12-31T14:58:32.9197388+02:00" start="2022-12-31T14:58:30.6234861+02:00" finish="2022-12-31T14:58:32.9364305+02:00" />
  <TestSettings name="default" id="b9b9b2c9-ee41-4056-8bab-0b3c722d7346">
    <Deployment runDeploymentRoot="_llaz-laptop_2022-12-31_14_58_32" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="d7ff78ea-1556-4bf5-89a3-487df2230b46" testId="bd09fc12-a765-ee3d-a1c0-3bb641b37717" testName="Tests.Map_WithConfiguredMappings_ShouldReturnObjectOfTargetType(sourceObject: Version1 { }, sourceType: typeof(Version1), targetType: typeof(Version5))" computerName="llaz-laptop" duration="00:00:00.0168553" startTime="2022-12-31T14:58:32.6450025+02:00" endTime="2022-12-31T14:58:32.6451295+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="d7ff78ea-1556-4bf5-89a3-487df2230b46" />
  </Results>
  <TestDefinitions>
    <UnitTest name="Tests.Map_WithConfiguredMappings_ShouldReturnObjectOfTargetType" storage="/home/llaz/git/playground/dotnet/transitive-mapper/bin/debug/net6.0/transitive-mapper.dll" id="bd09fc12-a765-ee3d-a1c0-3bb641b37717">
      <Execution id="d7ff78ea-1556-4bf5-89a3-487df2230b46" />
      <TestMethod codeBase="/home/llaz/git/playground/dotnet/transitive-mapper/bin/Debug/net6.0/transitive-mapper.dll" adapterTypeName="executor://xunit/VsTestRunner2/netcoreapp" className="Tests" name="Map_WithConfiguredMappings_ShouldReturnObjectOfTargetType" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="bd09fc12-a765-ee3d-a1c0-3bb641b37717" executionId="d7ff78ea-1556-4bf5-89a3-487df2230b46" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <Output>
      <StdOut>[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (64-bit .NET 6.0.11)
[xUnit.net 00:00:00.79]   Discovering: transitive-mapper
[xUnit.net 00:00:00.89]   Discovered:  transitive-mapper
[xUnit.net 00:00:00.90]   Starting:    transitive-mapper
[xUnit.net 00:00:01.10]   Finished:    transitive-mapper
</StdOut>
    </Output>
  </ResultSummary>
</TestRun>
Issafalcon commented 1 year ago

Hi @lawrence-laz - Thanks for pointing this out. I'm not sure exactly what's going on here. If you could raise another issue, that would be great. Also, I suspect that you're only seeing this with parameterized tests, judging by the screenshot? I may be wrong, but non-inlined parameterized tests behave unusually, so the fact you have a MemberData attribute, suggests to me that this is the likely culprit.

lawrence-laz commented 1 year ago

Hi @lawrence-laz - Thanks for pointing this out. I'm not sure exactly what's going on here. If you could raise another issue, that would be great. Also, I suspect that you're only seeing this with parameterized tests, judging by the screenshot? I may be wrong, but non-inlined parameterized tests behave unusually, so the fact you have a MemberData attribute, suggests to me that this is the likely culprit.

Reported as #40