search

DevExpress / testcafe-reporter-xunit

This is the xUnit reporter plugin for TestCafe.
https://testcafe.io
MIT License
10 stars 30 forks source link

Display unstable and screenshots as a separate tag rather than append to the title field #10

Closed vinayvennela closed 3 years ago

vinayvennela commented 3 years ago

Background:

When a test fails, then the screenshot path is appended to the name of the test When quarantine mode is enabled and if a test is unstable, then the tag '(unstable) ' is being appended to the name of the test

Both of the above look like a deviation of XML reports.

The suggestion is to have them added as a child tags inside the tag like we are adding

Current XML report -1: (Failure)


<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 91.0.4472.77 / macOS 10.15.7" tests="1" failures="1" skipped="0" errors="0" time="93.179" timestamp="Mon, 31 May 2021 11:49:16 GMT" >
  <testcase classname="dUTA Sample testcafe Tests" name="Open dUTA HomePage (screenshots: A possible lengthy path to screenshots)" time="40.762">
    <failure>
    <![CDATA[
      Stack Trace Goes here
    ]]>
    </failure>
  </testcase>
  <system-out>
  <![CDATA[
    Warnings (1):
    If any
  ]]>
  </system-out>
</testsuite>

Current XML Report - 2: (Unstable)

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 91.0.4472.77 / macOS 10.15.7" tests="1" failures="0" skipped="0" errors="0" time="93.179" timestamp="Mon, 31 May 2021 11:49:16 GMT" >
  <testcase classname="dUTA Sample testcafe Tests" name="Open dUTA HomePage (unstable)" time="40.762">
    <screenshots>path/to/screenshots</screenshots>
  </testcase>
  <system-out>
  <![CDATA[
    Warnings (1):
    If any
  ]]>
  </system-out>
</testsuite>

Proposed XML Report - 1 (Failed)

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 91.0.4472.77 / macOS 10.15.7" tests="1" failures="1" skipped="0" errors="0" time="93.179" timestamp="Mon, 31 May 2021 11:49:16 GMT" >
  <testcase classname="dUTA Sample testcafe Tests" name="Open dUTA HomePage" time="40.762">
    <failure>
    <![CDATA[
      Stack Trace Goes here
    ]]>
    </failure>
    <screenshots>path/to/screenshots</screenshots>
  </testcase>
  <system-out>
  <![CDATA[
    Warnings (1):
    If any
  ]]>
  </system-out>
</testsuite>

Proposed XML Report - 2 (Unstable)

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 91.0.4472.77 / macOS 10.15.7" tests="1" failures="0" skipped="0" errors="0" time="93.179" timestamp="Mon, 31 May 2021 11:49:16 GMT" >
  <testcase classname="dUTA Sample testcafe Tests" name="Open dUTA HomePage" time="40.762">
    <unstable>true</unstable>
  </testcase>
  <system-out>
  <![CDATA[
    Warnings (1):
    If any
  ]]>
  </system-out>
</testsuite>
vinayvennela commented 3 years ago

Will be happy to contribute and create MR

wentwrong commented 3 years ago

I'm closing this issue since this was already discussed in the context of #2. Also I'm not sure that the xUnit format allows specifying such tags as screenshots and unstable and I'm afraid that the consumers of this format aren't aware of these tags either, so I think it's better to stick to the current implementation.

You can create your own fork of this reporter, or use the existing fork posted in the issue above (where all of the described problems were fixed). Also if you're using this reporter along with Jenkins, I recommend that you look at the Jenkins reporter instead (see Introducing the TestCafe Jenkins Plugin).

vinayvennela commented 3 years ago

Appending screenshots path to the test name isn't just right way to do. I'm sure xUnit standard doesn't supporting appending customs tags and paths to the test name either, but we are doing it.

Every consumer of this reporter must have written the same logic to strip off the screenshots paths and unstable tags before parsing the xml. XMLs are indeeded meant for machines.

This is the reporter officially supported by testcafe and at some point we need to fix this.

Agreed that users may not be aware of these new tags. Let's update the document and bump the major version.

wentwrong commented 3 years ago

The current approach with appending the (unstable) tag and the screenshot path to the name of a test seems to be better as it doesn't violate the xUnit format. After all, the name attribute of the testsuite and testcase tags remains to be just a string.

I agree that users are still forced to parse this information from the test name but this is inevitable since we want to stay within the xUnit format, which simply doesn't provide special tags/attributes for such information. Thank you for your proposal but we do not want to change the xUnit format by introducing new tags.