After generateReport is called, the xunit style differs from the old TestNG way of reporting

[rarebreed]

I ran the full sm-cli-acceptance suite last night and once it completed the generated XML file was huge (72 MB). Also, the name of the test suite and the other attributes were wrong:

 <testsuite name="CLI: Help Tests" tests="0" failures="0" time="0.0" skipped="0">

Clearly, that isn't correct.

Worse, when this xunit file is POST'ed, it doesn't seem to go through. Or rather, there is no record of the file being processed by the XUnit Importer plugin in polarion.

[rarebreed]

Ok, I see what's going on. I only ever tested before on a suite.xml that only had one class inside it. So while designing the "algorithm" to walk through the suite, that worked.

Problem is, if the suite contains multiple classes, TestNG defines a suite as a map of class -> result. So what I really need to do is accumulate the number of passes, fails, and skips from all the results in the map

[rarebreed]

On reflection, this isn't necessarily wrong (other than the time stamp being wrong and the way it determines total number of tests), but it is different in behavior from the TestNG way of reporting a result.

In xunit, you send a suite of suites:

<testsuites>
  <testsuite name="CLI: Register Tests" tests="7" failures="0" time="312.0" skipped="2">
    ... <testcase> are listed here
  </testsuite>
  <testsuite name="CLI: Interoperability Tests" tests="1" failures="0" time="13.0" skipped="0">
  ... <testcase> are listed here etc etc
  </testsuite>
  ...
</testsuites>

In the TestNG result format, we just had one huge suite which was essentially the name of the suite.xml file we ran against. If we want to duplicate the testng style, it would have to look something like this:

<testsuites>
  <testsuite name="sm-cli-acceptance" tests="1000" failures="20" time="13.0" skipped="30">
    ... all the <testcase> from every "CLI: XXXXX Tests" would go here
  </testsuite>
</testsuites>

Either way, a bit of additional work needs to be done:

• Need to add a new parameter which would be the name of:
  • the \<testsuites> if doing xunit way
  • the name of \<testsuite> if doing the testng way
• If doing the testng way, I need to concatenate all the \<testsuite> results into one giant one
  • That means summing up all the success/fail/error/skips
  • every \<testcase> will belong to the one and only \<testsuite>
• Fix the time stamp to divide by 1000 since time is in milliseconds
• Determine how to correctly count the number of tests executed

The last point bears some discussion. The way the code determines the total number of fails is to use the

result.getTestContext().getFailedTests().size()  // total failed
result.getTestContext().getAllTestMethods().length  // total number of tests executed

The problem is that this is not taking into account data driven tests, only the actual methods. For example, let's say Foo was a data driven test that ran 10 permutations of arguments. While executing 9 of the permutations passed, but one failed. In this case, Foo failed, so getFailedTests().size() would return 1. Also, just calling the getAllTestMethods().length would also return 1.

Should this really reflect that we ran 10 methods with 9 passes and 1 failure?

[rarebreed]

I believe this has mostly been fixed with the latest code, though it still diverges somewhat. With the latest code, each class is its own testsuite. Although polarize now counts each iteration of the method + args permuation to count towards the total number of tests, it appears that Polarion is not showing anything different.

[rarebreed]

Since the Polarion TestRun is basically just a collection of test records, the run in polarion will be similar to testng in that every iteration of every method will fall under a single "artifact" (ie a TestRun in polarion, a TestSuite in testng)