Reporter output doesn't output anything even if we have data in reporter-output.html file

[satish88]

TestNG Version : 6.8.7 (reproducible in 6.9.8 also) useDefaultListeners : true (use default listeners) One of which is org.testng.reporters.jq.Main

Now log anything from test cases using Reporter.log. For reproduction, I used 2 test cases with retry mechanism in place using IRetryAnalyzer.

I am failing both the tests intentionally. When I see the reports, in target/surefire-reports/index.html (produced by org.testng.reporters.jq.Main) doesn't show anything in Reporter output section, sometimes it shows entry for one test case and not for the other. While data is always there in target/surefire-reports/old/<suite_name>/reporter-output.html.

While debugging, I found when we create report in org.testng.reporters.jq.Main class. It goes to following method in ReporterPanel to get data from Reporter:

public String getContent(ISuite suite, XMLStringBuffer main) {
    XMLStringBuffer xsb = new XMLStringBuffer(main.getCurrentIndent());
    for (ITestResult tr : getModel().getAllTestResults(suite)) {
      List<String> lines = Reporter.getOutput(tr);
      if (! lines.isEmpty()) {
        xsb.push(D, C, "reporter-method-div");
        xsb.addRequired(S, Model.getTestResultName(tr), C, "reporter-method-name");
        xsb.push(D, C, "reporter-method-output-div");
        for (String output : lines) {
          xsb.addRequired(S, output, C, "reporter-method-output");
        }
        xsb.pop(D);
        xsb.pop(D);
      }
    }
    return xsb.toXML();
  }

In this, Reporter is coming as null, due to which it doesn't get any data. I also tried to disable all reporters except this to give it first chance to access Reporter, but no luck.

[juherr]

If I understand you well, Reporter.getOutput(tr) should just return an empty list instead of null.

Could you try if it fixes your problem? Could you provide a full sample too?

[satish88]

Working on creating a sample

[satish88]

Hi Julien, When i stripped off extra code and kept only one test case in one class, it worked as expected. I also tried to use several test classes but could not reproduce the scenario.

Having said that, I am pretty sure it happened several times when I filed this bug. Is there any debug mode when we can get detailed logs from testng if it occurs again in any situation. Because It would not be possible to share the whole codebase due to company restriction.

[juherr]

Are you using parallel when you observe the issue? Do you observe the issue from your IDE too or just via maven?

You can activate debug mode by activating log level:

• <suite name="Suite1" verbose="10"> in the suite file, or
• via the level of verbosity in surefire.

[satish88]

Yes, We are using parallel and I doubt that may be the main reason. Because, when I was facing this issue, even that time it was not always reproducible. We are invoking tests with maven surefire plugin

[juherr]

We had sometime random issues with parallel tests. That's why I think it is related.

Did you reproduce with the latest testng ?

[satish88]

I checked with 6.9.8 . It behaved in the same manner as 6.8.7

[juherr]

Could you try with 6.9.10 (latest on maven central) or later (available on bintray) ?