Issue with XMLReporter - <system-out> field contains empty <![CDATA[]]>

misiekstewa commented 5 years ago

TestNG Version

6.14.3

Expected behavior

Inside the TEST-mytestname.xml when it's running with JUnit 4

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" hostname="mw" name="tests.external_frameworks.PetclinicTests" skipped="0" tests="3" time="0.155" timestamp="2019-03-08T08:48:00">
  <properties/>
  <testcase classname="tests.external_frameworks.PetclinicTests" name="validateSecurityLog" time="0.001" />
  <system-out><![CDATA[Starting process: /home/users/mmmm/builds/jdk-hs-8u171-linux-x64/bin/java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b25, mixed mode)
Starting process: /home/users/mmmm/builds/jdk-hs-8u171-linux-x64/bin/java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b25, mixed mode)
Starting process: /opt/wawawa/java/jdk-1.8.0/bin/java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
Starting process: hostname
.
.
.
]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Actual behavior

Inside the TEST-mytestname.xml when it's running with TestNG

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" hostname="mw" name="tests.external_frameworks.PetclinicTests" skipped="0" tests="3" time="0.155" timestamp="2019-03-08T08:48:00">
  <properties/>
  <testcase classname="tests.external_frameworks.PetclinicTests" name="validateSecurityLog" time="0.001" />
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Is the issue reproductible on runner?

[ ] Shell
[ ] Maven
[x] Gradle
[ ] Ant
[ ] Eclipse
[ ] IntelliJ
[ ] NetBeans

Test case sample

I'm not sure if it's an issue or expected behaviour but when I'm running the same test with JUnit in the .xml report I'm able to see the system output, which is executed in static blocks before the test. So I mean everything which is appearing after:

> Task :Frameworks:test
Starting process 'Gradle Test Executor 1'. Working directory: /home/users/Projects/automation/Frameworks Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.el6_7.x86_64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=IE -Duser.language=en -Duser.variant -ea -cp /home/users/mmmm/.gradle/caches/4.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'

Gradle Test Executor 1 STANDARD_OUT

but if I update annotations to TestNG annotations and update mu build.gradle file with

test {
    testLogging.showStandardStreams = true
    useTestNG() {
        useDefaultListeners = false
    }
}

then standard output is visible in the console but it's not saved in the .xml report file.

If that is a bug (not only issue with bad configuration) then I'll try to implement some small test which will be reproducing this.

krmahadevan commented 5 years ago

I don't think that the xml reports that TestNG generates ever captured the sys out and sys error in the reports.

ping @cbeust @juherr WDYT ?

juherr commented 5 years ago

If possible, I think TestNG should have the same result than JUnit. But I don't know what could be the technical impacts

krmahadevan commented 5 years ago

@juherr - I think it would be a challenge when trying to implement this for parallel execution, because the underlying PrintStream within System class dont seem to be thread safe.

juherr commented 5 years ago

Right, maybe the feature can be only activated when parallel is not activated.

Not a priority issue IMO.

UrsLange commented 5 years ago

I'd like to extend on this issue. I think the report, generated by org.testng.reporters.JUnitReportReporter should at least show the output, previously sent to org.testng.Reporter. This should be possible by writing the content of Reporter.getOutput(currentTestResult) to the <system-out> tag in the Report. Compared to handling sysout and syserr, this sould be relatively easy to implement. Thank you!

UrsLange commented 5 years ago

@krmahadevan May I ask you kindly, for feedback on my request? Implementing this would be a considerable help for my project. Thank you!

krmahadevan commented 5 years ago

@UrsLange - I don't want to club your ask in this issue, because this issue talks about capturing the sys out output, and you are asking for Reporter.log() contents. Please open up a new issue for your ask and I will try to get to it.

UrsLange commented 5 years ago

@krmahadevan Thank you for your response. I created #2124 for the matter.

testng-team / testng