search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

Different results in the reports #1830

Open diegolovison opened 6 years ago

diegolovison commented 6 years ago

TestNG Version

6.14.3

Description

We have 3 tests:

We have the following configuration in pom.xml

<excludedGroups>windows</excludedGroups>

In this case, 2 tests are supposed to be running and 1 should be skipped.

In the console we have:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.176 sec - in TestSuite

In JUnitXMLReporter ( https://github.com/diegolovison/testng-reporter/blob/master/target/surefire-reports/Surefire%20suite/Surefire%20test.xml ) we have:

ignored="1"
<testcase name="a" ignored

In JUnitReportReporter ( https://github.com/diegolovison/testng-reporter/blob/master/target/surefire-reports/junitreports/TEST-com.github.diegolovison.testngreporter.FooTest.xml ) we have:

skipped="0"

No mention about FooTest:a

Expected behavior

I would like to have the same result in console, JUnitXMLReporter and JUnitReportReporter

Actual behavior

Different value in all reports

Is the issue reproductible on runner?

Test case sample

https://github.com/diegolovison/testng-reporter

krmahadevan commented 6 years ago

@diegolovison - Can you please check if this library that I built works for you ? https://github.com/RationaleEmotions/junitreport

diegolovison commented 6 years ago

Hi @krmahadevan

Well, no. I generated the report as you can see here https://github.com/diegolovison/testng-reporter/blob/rationale-emotions/target/surefire-reports/Surefire%20suite/Surefire%20test.xml it does not show what I would like to see

The "junitreports" https://github.com/diegolovison/testng-reporter/tree/rationale-emotions/target/surefire-reports/junitreports shows what I am expecting.

What is your opinion?

krmahadevan commented 6 years ago

@diegolovison - I didn't quite understand. So are you saying that junitreports library that I shared works and solves your purpose?

diegolovison commented 6 years ago

@krmahadevan, no. In your report, where I can see that the method c in FooTest skipped?

  <testcase name="c" classname="com.github.diegolovison.testngreporter.FooTest">
    <skipped/>
  </testcase>
krmahadevan commented 6 years ago

@diegolovison - I am not sure if skipped is a recognized status in the JUnit world. My library refers to the JUnit.xsd available here https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd

diegolovison commented 6 years ago

Well, the info above related with FooTest:c comes from JUnitReportReporter as you can see here: https://github.com/diegolovison/testng-reporter/blob/rationale-emotions/target/surefire-reports/junitreports/TEST-com.github.diegolovison.testngreporter.FooTest.xml

What I would like to address in this issue is keep console, JUnitXMLReporter and JUnitReportReporter consistent. Today they have different results what cause confusion.

krmahadevan commented 6 years ago

@diegolovison - Let me see what I can do. Do you think you could help quickly share a sample that can be used to reproduce the problem ? That way its easy to get to the bottom of this issue.

diegolovison commented 6 years ago

@krmahadevan yes. It is described on the issue :) https://github.com/cbeust/testng/issues/1830#issue-332075404 There is a test case here https://github.com/diegolovison/testng-reporter If you have any question, please let me know