spekt / xunit.testlogger

XUnit logger for vstest platform
MIT License
73 stars 14 forks source link

Wrong test name when using custom framework #57

Open D0tNet4Fun opened 1 week ago

D0tNet4Fun commented 1 week ago

Hello,

I found an issue with the Xunit logger when using it with my Xunit-based custom test framework. Let me give you a bit of context:

This test framework allows using a class to define a test case. For example this is a test case having 6 steps:

[TestCase("TC0001")]
public class DemoTestCase
{
    [Summary("Login to web site")] public void LoginToWebsite() { }

    [Precondition(1, "The web site must be deployed on the web server")] private void DeployWebSite() { }     
    [Precondition(2, "The web site must be accessible via HTTPS")] private void EnableHttps() { }
    [Input(1, "Access the web site")] private void AccessWebSite() { }
    [Input(2, "Enter user name and password")] private void EnterUserNameAndPassword() { }
    [Input(3, "Click the login button")] private void Login() { }
    [ExpectedResult(3, "The user should be logged in")] private void VerifyUserIsLoggedIn() { }
}

In the discovery phase this shows up as a single test case named "Login to web site". When run, this test case produces 7 results: one for each of the 6 steps and another one for the Summary. Here's what this looks like in the VS Test Explorer:

The issue I found is that the test report contains the 7 test results as expected, but they all have the same name "Login to web site":

<test name="Login to web site" type="Automation.TestFramework.Tests.DemoTestCase" method="LoginToWebsite" time="0.0009059" result="Pass">
  <traits />
</test>

By contrast, the trx test report contains the correct names as provided by my framework:

<Results>
  <UnitTestResult testName="[1/6] [Precondition] 1. The web site must be deployed on the web server" />
  <UnitTestResult testName="[3/6] [Input] 1. Access the web site" />
  <UnitTestResult testName="[4/6] [Input] 2. Enter user name and password" />
  <UnitTestResult testName="[5/6] [Input] 3. Click the login button" />
  <UnitTestResult testName="Login to web site" />
  <UnitTestResult testName="[6/6] [Expected result] 3. The user should be logged in" />
  <UnitTestResult testName="[2/6] [Precondition] 2. The web site must be accessible via HTTPS" />
</Results>
<TestDefinitions>
  <UnitTest name="Login to web site" storage="d:\work\automation.testframework.git\automation.testframework.tests\bin\debug\net8.0\automation.testframework.tests.dll">
    <Execution id="b69d9c91-53d7-4d4c-bbd3-688099c61c1f" />
    <TestMethod codeBase="d:\Work\Automation.TestFramework.git\Automation.TestFramework.Tests\bin\Debug\net8.0\Automation.TestFramework.Tests.dll" adapterTypeName="executor://xunit/VsTestRunner2/netcoreapp" className="Automation.TestFramework.Tests.DemoTestCase" name="LoginToWebsite" />
  </UnitTest>
</TestDefinitions>

The trx logger makes this difference between the test case name and the test result name, whereas the xunit logger does not.

Could you update the logger to use the test result name in this case, please?

D0tNet4Fun commented 1 week ago

I've done some digging and found out that the data I need is available on the TestResultInfo, although as internal property:

XunitXmlSerializer.CreateTestElement(TestResultInfo result)

result.DisplayName = "Login to web site"
result.TestCaseDisplayName = "Login to web site"
result.TestResultDisplayName = "[1/6] [Precondition] 1. The web site must be deployed on the web server"

I modified this code (in a way that is not worthy of a PR) and got this beautiful test report:

<test name="[1/6] [Precondition] 1. The web site must be deployed on the web server" type="Automation.TestFramework.Tests.DemoTestCase" method="LoginToWebsite" time="0.0009317" result="Pass">
  <traits />
</test>
<test name="[2/6] [Precondition] 2. The web site must be accessible via HTTPS" type="Automation.TestFramework.Tests.DemoTestCase" method="LoginToWebsite" time="0.0000170" result="Pass">
  <traits />
</test>
...