Include the `file` attribute to the test report

[rob4629]

I'm using CircleCi to run our regression tests, but we're unable to use the Split-By Test Timings option because the results output doesn't contain a file= attribute .

CircleCI uses the file attribute to associate the timing data of the class names in each file... without that it defaults to splitting by name. This results in unbalanced regression runs, slowing down our CI process.

[SergeyShurygin]

HI @rob4629. Could you please provide a sample output with the file attribute? I examined several examples of xsd schemas for the xUnit report format (e.g. https://gist.github.com/erikd/4192748) and didn't find any file attribute

[rob4629]

Hi @SergeySeleznyov, sorry for not providing an example 🤦. Here's an example of our Mocha tests output:

<testsuites name="Mocha Tests" time="6.735999999999989" tests="555" failures="7">
  <script>
    ***
  </script>
  <testsuite name="service log" timestamp="2020-04-21T21:53:05" tests="6" file="/home/circleci/change/payment_service/test/lib/braintree/events.test.js" failures="0" time="0.006">
    <testcase name="Braintree event handling logWebhookEvent service log includes extra log data passed to it" time="0.002" classname="includes extra log data passed to it"> </testcase>
    <testcase name="Braintree event handling logWebhookEvent service log includes webhook context" time="0.001" classname="includes webhook context"> </testcase>
    <testcase name="Braintree event handling logWebhookEvent service log includes gateway" time="0.001" classname="includes gateway"> </testcase>
    <testcase name="Braintree event handling logWebhookEvent service log includes gateway event id" time="0.001" classname="includes gateway event id"> </testcase>
    <testcase name="Braintree event handling logWebhookEvent service log includes event data" time="0.001" classname="includes event data"> </testcase>
    <testcase name="Braintree event handling logWebhookEvent service log includes gateway event timestamp" time="0" classname="includes gateway event timestamp"> </testcase>
  </testsuite>
</testsuites>

As you can see, the file attribute provides detail of the test file's location.

[rob4629]

Looking further in to the TestCafe output file... it looks like there's only 1 <testsuite> </testsuite> section, rather than having

<testsuites name="X" time="XX.XXX" tests="X" failures="X">
  <testsuite name="X1" tests="X" failures="0" skipped="0" errors="0" time="XX.XXX" timestamp="2020-04-21T21:53:05" file="XX.js">
    <testcase name="XX1" time="0.002" classname="XX"> </testcase>
    <testcase name="XX2" time="0.002" classname="XX"> </testcase>
    <testcase name="XX3" time="0.002" classname="XX"> </testcase>
  </testsuite>
  <testsuite name="X2" tests="X" failures="0" skipped="0" errors="0" time="XX.XXX" timestamp="2020-04-21T21:53:05" file="XX.js">
    <testcase name="XX1" time="0.002" classname="XX"> </testcase>
    <testcase name="XX2" time="0.002" classname="XX"> </testcase>
    <testcase name="XX3" time="0.002" classname="XX"> </testcase>
  </testsuite>
</testsuites>

[miherlosev]

Hi @rob4629

Thank you for the example. We will check the possibility of adding the file attribute to a test report.

[davidmontoyago]

Hi @miherlosev, any thoughts on adding this feature? I'm dealing with the same issue. Here's the CircleCI issue where they describe what the expected attributes are.

I'd be happy to submit a quick PR if the top contributors agree with the change. There is apparently not much of a standard when it comes to xUnit.

[miherlosev]

Hi @davidmontoyago

TestCafe's xUnit reporter output contains only one <testsuite> section.

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 84.0.4147.135 / Windows 10, Firefox 79.0 / Windows 10" tests="4" failures="0" skipped="0" errors="0" time="6.756" timestamp="Tue, 25 Aug 2020 07:41:44 GMT" >
  <testcase classname="Fixture1" name="test1" time="1.214">
  </testcase>
  <testcase classname="Fixture1" name="test2" time="0.674">
  </testcase>
  <testcase classname="Fixture2" name="test3" time="3.552">
  </testcase>
  <testcase classname="Fixture2" name="test4" time="3.437">
  </testcase>
</testsuite>

As far as I understand the file attribute can contain only one file path. TestCafe can execute few files with tests. It looks like we cannot support the file attribute for the xUnit report.

[toan-tran-hrs]

I'm pretty sure that this issue can be solved and want to provide my solution. Instead of only using name in the reportFixtureStart method, we can use a new attribute called currentFixture which contains both of fixture's name and path. Then we can safely use it as a better replacement for currentFixtureName.

If this issue is solved, it will allow us to use the report with CircleCI's timing strategy for parallel execution.

[rob4629]

If you have a PR, I'd love to review it (as this is also something I've had on my backlog to resolve for about 18months now)

[toan-tran-hrs]

I don't have the permission for creating remote branch now but I have one locally and it's ready to be pushed.

[rob4629]

You should be able to fork this repo, then create a branch (of your forked repo), and create a PR from that.

[viktoria2506]

Thank you for your contribution to TestCafe. We will review this PR.

[gromran]

nice, the "file" attribute broke jenkins evaluation.... Attribut 'file' darf nicht in Element 'testcase' vorkommen.

[AlexKamaev]

@gromran I installed Jenkins locally and did not face any issue with the file attribute. A possible cause of the issue is that you are using not the latest Jenkins version with an outdated JUnit plugin (in my local Jenkins the JUnit plugin version is 1.56). Please try updating the plugin or try running tests on a new local Jenkins instance to check my assumption.

[rob4629]

Closing because this has been implemented