Number of failures and errors in `testsuite` element

[na9amura]

Background

I tried to use AWS CodeBuild and its reporting feature. But I received error message below about report file interpretation.

Error in UPLOAD_ARTIFACTS phase: [testcafe-reports: [error processing report: [InvalidInputException: Test summary: status count cannot be negative]]]

Question

When test report include failures, those case would be counted in both failures and errors in testsuite element. Is it correct behavior?

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 96.0.4664.110 / macOS 10.15.7" tests="1" failures="1" skipped="0" errors="1" time="2.451" timestamp="Fri, 17 Dec 2021 10:12:26 GMT" >
  <testcase classname="..." name="test (screenshots: .../errors/1.png)" time="2.428">
    <failure>

According to this definition, failures and errors should not be overlapped (I'm not confident about reliability of this document). https://github.com/windyroad/JUnit-Schema/blob/d6daa414c448da22b810c8562f9d6fca086983ba/JUnit.xsd#L186-L195

[AlexKamaev]

It seems that the behavior described in the document you provided is correct. However, I did not find any official xunit specifications that describe the difference between errors and failures. At this moment, TestCafe does not distinguish between errors and failures. We'll take this as a suggestion, but I cannot give any estimates on when it will be implemented.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[github-actions[bot]]

We're closing this issue after a prolonged period of inactivity. If it still affects you, please add a comment to this issue with up-to-date information. Thank you.