search

toeb / googletest

Automatically exported from code.google.com/p/googletest
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

When --gtest_filter is specified, the XML output shouldn't contain info about tests that aren't run. #141

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Currently, when --gtest_filter is specified, the XML output still contains
info about all tests.  Those not run are marked as status="notrun".  This
makes it difficult to find the interesting info.  There should be an option
(maybe we should even abandon the current behavior) to include only tests
actually executed in the XML report.

Original issue reported on code.google.com by zhanyong...@gmail.com on 14 Apr 2009 at 6:03

GoogleCodeExporter commented 9 years ago 
There are several complications when implementing this.  They aren't
insurmountable, but worth noting:

- How should disabled tests be treated?
- What about the --gtest_also_run_disabled_tests flag?  How should it
affect the XML report?
- How should the test sharding affect the XML report?

My idea:

- The XML report should include all tests that match the test filter,
regardless of them being disabled or not.
- The XML report should not include any tests other than the above.

That means:

- The "disabled" status of a test doesn't affect whether it shows up
in the report: it just affects the "disabled" attribute of the test.
- The --gtest_also_run_disabled_tests flag doesn't affect which tests
are in the report either: it just affects whether a test is "run" or
"notrun".
- Test sharding doesn't affect which tests are in the report: it just
affects whether a test is "run" or "notrun".

In other words, the test filter is the only thing that may affect
which tests end up in the report.

I think this should have minimal impact on existing systems that
consume the XML report.

Original comment by zhanyong...@gmail.com on 30 Aug 2010 at 5:45

GoogleCodeExporter commented 9 years ago

Original comment by w...@google.com on 27 Sep 2010 at 10:04

GoogleCodeExporter commented 9 years ago 
Hi All,

When would this issue be fixed?We are also facing a similar problem where the 
Jenkins Junit report is not able to generate appropriate report.

Regards
Prashant Katti

Original comment by prashant.katti on 12 Apr 2011 at 6:54

GoogleCodeExporter commented 9 years ago 
status="notrun" is not useful for Jenkins/Hudson while parsing the results and 
marking test as skipped. It requires <testcase> tag with nested <skipped> tag 
as below

<testcase name="testIgnorable>
    <skipped/>
</testcase>

Original comment by pvshew...@gmail.com on 12 Apr 2011 at 7:05

GoogleCodeExporter commented 9 years ago 
I second this (comment 4).

The change would be a small addition in 
XmlUnitTestResultPrinter::OutputXmlTestInfo

An additional tag should not break anything in existing environments (e.g. 
tools relying on testcases with attribute status="notrun") and it would be a 
gread benefit for tools relying on the additonal skipped tag.

Original comment by housemaister on 24 Aug 2011 at 8:11

GoogleCodeExporter commented 9 years ago 
The tests that have been filtered out as not part of the current shard should 
also have no records in the XML.

Original comment by vl...@google.com on 26 Sep 2011 at 10:38

GoogleCodeExporter commented 9 years ago 
I second comment 4 too, since the output format is JUnit xml, <skipped/> is the 
way to do it.

Original comment by quique.l...@gmail.com on 22 Nov 2011 at 11:41

GoogleCodeExporter commented 9 years ago 
+1 for comment 4.  Both for disabled and filtered out tests.

Original comment by asmu...@gmail.com on 13 Jan 2012 at 2:12

GoogleCodeExporter commented 9 years ago 
We are having a similar problem where our disabled tests are showing up as 
Passed. They should not be since they were not run. +1 for comment 4.

Original comment by rdevera...@gmail.com on 22 Feb 2012 at 8:06

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I wrote a patch that filter tests on xml output. It matches the same filter 
pattern behavior printed on the console. I'm attaching a filtered xml report. 
I'm planing to submit the code, but I was wondering whether to implement a 
solution for disabling test and test sharding... or just leave it like that. 
Any suggestion/request? 

Thanks,
Armando.

Original comment by afonseca...@gmail.com on 7 May 2012 at 4:57

Attachments:

GoogleCodeExporter commented 9 years ago 
I use the following modification to generate xml that allows Jenkins to 
correctly parse skipped tests.

    Modify xml output, so that Jenkins correctly reports skipped tests

    Jenkins expects the <skipped> tag.  It ignores the status attribute.

diff --git a/gtest/gmock-gtest-all.cpp b/gtest/gmock-gtest-all.cpp
index f8dc5a6..f489d97 100644
--- a/gtest/gmock-gtest-all.cpp
+++ b/gtest/gmock-gtest-all.cpp
@@ -4587,14 +4587,16 @@ void 
XmlUnitTestResultPrinter::OutputXmlTestInfo(::std::ostream* stream,
           << "\" time=\""
           << FormatTimeInMillisAsSeconds(result.elapsed_time())
           << "\" classname=\"" << EscapeXmlAttribute(test_case_name).c_str()
-          << "\"" << TestPropertiesAsXmlAttributes(result).c_str();
+          << "\"" << TestPropertiesAsXmlAttributes(result).c_str()
+          << ">\n";
+
+  if (!test_info.should_run()) {
+    *stream << "      <skipped type=\"skipped\" message=\"skipped\"/>\n";
+  }

-  int failures = 0;
   for (int i = 0; i < result.total_part_count(); ++i) {
     const TestPartResult& part = result.GetTestPartResult(i);
     if (part.failed()) {
-      if (++failures == 1)
-        *stream << ">\n";
       *stream << "      <failure message=\""
               << EscapeXmlAttribute(part.summary()).c_str()
               << "\" type=\"\">";
@@ -4607,10 +4609,7 @@ void 
XmlUnitTestResultPrinter::OutputXmlTestInfo(::std::ostream* stream,
     }
   }

-  if (failures == 0)
-    *stream << " />\n";
-  else
-    *stream << "    </testcase>\n";
+  *stream << "    </testcase>\n";
 }

 // Prints an XML representation of a TestCase object

Original comment by sproha...@gmail.com on 14 Jul 2012 at 7:23