search

comipayan / googletest

Automatically exported from code.google.com/p/googletest
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

XML output should match the junitreport Ant task #114

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Currently, the XML output mode for the --gtest_output switch produces a single 
XML file 
containing all testsuites and testcase results.  Such a file cannot be 
correctly parsed by some CI 
servers, namely Hudson.  Issue #77 erroneously blames gtest's XML formatting 
for this bug, but 
the actual cause of this problem is that Hudson expects to parse JUnit results 
as one XML file per 
class.

I wrote a perl script to break apart the XML output generated by gtest into 
individual files, and 
called it from ant in hudson.  Doing this fixes the issue, which serves as a 
workaround (I've also 
attached the script to this report).

It would be nice if the gtest_output switch could perhaps include support for 
writing individual 
XML output files.  Perhaps this could be done either with 
"gtest_output:xmldir", or by parsing the 
argument given here to see if it refers to an existing directory (though this 
would probably be 
confusing).

What steps will reproduce the problem?
1. Generate XML output using --gtest_output:xml
2. Configure hudson project to parse JUnit results with this file
3. Hudson then complains that "None of the test reports contained any result"

What version of the product are you using? On what operating system?
1.2.1

Original issue reported on code.google.com by njreiman@gmail.com on 10 Feb 2009 at 1:45

Attachments:

GoogleCodeExporter commented 9 years ago 
Since it's inefficient to generate many XML output files for each test program, 
we
prefer the issue to be fixed on the Hudson side.

Original comment by zhanyong...@gmail.com on 10 Feb 2009 at 4:20

GoogleCodeExporter commented 9 years ago 
We are using Hudson as our CI server as well, like the original poster has 
said. Since 
JUnit outputs multiple XML files, in order to allow broader compatibility with 
existing 
tools, I believe gtest should also offer an similar option to output multiple 
XML files.

I understand it's inefficient at times to generate multiple XML output files 
for each 
test (class), but as long as it is provided as an additional option 
(--gtest_output 
...), we can satisfy both camps of people.

Although a workaround is available, we prefer to keep our build system clean 
and not 
have to depend on Perl or other translation facilities.

Is it possible to raise the priority of this issue?

Original comment by charles....@gmail.com on 26 Feb 2009 at 10:01

GoogleCodeExporter commented 9 years ago 
Charles found this thread: 
http://www.nabble.com/Re:-Getting-Hudson-to-understand-
Googletest-XML--p22182483.html

According to it, if we change the root element from "testsuite" to 
"testsuites", the 
output will work with Hudson.

Google Test was designed to match our internal tools which expect the
XML report to match the format of those produced by the "junit" Ant
task, which has "testsuite" as the root element.

The format Hudson uses corresponds to the "junitreport" Ant task,
which has "testsuites" as the root element.

Unfortunately, if we change Google Test's format, we will break the
build system integration at Google.  I see two possible solutions:

1. Make Hudson support the "junit" format in addition to the
"junitreport" format.
2. Make Google Test support generating both formats, e.g. we can
continue to use --gtest_output=xml to select the current format, and
use --gtest_output=xmlreport to select the junitreport-style format.

We can work on #2, but given the limited resources, we may not be able
to get to it soon.  Therefore, I'm asking Charles to talk to the Hudson guys to
see if they can get #1 done quickly.  Meanwhile, I'm changing the priority of 
this 
issue to Medium.

Original comment by zhanyong...@gmail.com on 9 Mar 2009 at 5:25

GoogleCodeExporter commented 9 years ago 
As long as you have configured Hudson to do the jobs with an Ant build script, a
simple workaround for renaming the root element of the Googletest XML log is to
append a <replace> task to the call of the executable. 
Let us assume that you have a build target called 'run_test' that simply calls 
your
googletest tests called 'GoogleTests.exe'. Then the following XML snipplet will
execute the test and rename the root element of the created XML log so that 
Hudson
will render the junitreport properly:

<target name="run_test" description="Runs the googletests and translates the 
log">
  <exec executable="GoogleTests.exe">
     <arg value="--gtest_output=xml:GoogleTests.xml" />
  </exec>
  <replace file="GoogleTests.xml" token="testsuite tests" value="testsuites tests" />
</target>

However, this is just a workaround for the 'junitreport'. In the end, I would
appreciate a solution implementing njreiman suggestions (additional output 
options).

Original comment by rup...@gmail.com on 18 Mar 2009 at 11:49

GoogleCodeExporter commented 9 years ago 
Hmm -- that ant <replace> task is definitely cleaner than calling out to perl.  
But 
again, I would still prefer separate XML reports for each class, as that better 
reflects JUnit's behavior and is easier (for humans) to parse.

I think the --gtest_output=xmlreport option is probably the best suggestion 
we've had 
so far.  If I have some time in the next weeks, I'll try to write a patch for 
this, 
since the gtest team understandably has bigger bugs to squash first.

Original comment by njreiman@gmail.com on 18 Mar 2009 at 11:56

GoogleCodeExporter commented 9 years ago 
I talked with more people at Google, and we decided that we will change our 
internal
format to match the junitreport Ant task.  Therefore the plan is to change the
behavior of --gtest_output=xml to generate junitreport.  We won't be adding a 
new
option --gtest_output=xmlreport.  At least not yet.

Original comment by zhanyong...@gmail.com on 20 Mar 2009 at 12:27

GoogleCodeExporter commented 9 years ago 
Editing the subject to match what we plan to do.

Original comment by zhanyong...@gmail.com on 23 Mar 2009 at 7:55

GoogleCodeExporter commented 9 years ago 
Implemented in the trunk.

Original comment by zhanyong...@gmail.com on 24 Mar 2009 at 9:06

GoogleCodeExporter commented 9 years ago 
Which version do you mean by trunk? I am using gtest 1.3.0, but I am having 
trouble
parsing the generated XMLs with Hudson?

Original comment by kbatra.a...@gmail.com on 16 Apr 2009 at 5:09

GoogleCodeExporter commented 9 years ago 
The "trunk" means the development branch in the source code repository, which 
can be 
accessed here:

http://code.google.com/p/googletest/source/browse/#svn/trunk

Or if you prefer, you can just wait until version 1.3.1.

Original comment by njreiman@gmail.com on 16 Apr 2009 at 7:18

GoogleCodeExporter commented 9 years ago 
When will be the 1.3.1 version be available?

Original comment by anubh...@gmail.com on 27 Aug 2009 at 3:11

GoogleCodeExporter commented 9 years ago 
Since this is still not released 5 months later, I've attached an XSLT file 
that does
the fix:

Which can be applied in ant by doing something like:

<xslt in="test_detail.xml" out="TEST-results.xml" style="fix-testsuites.xsl"/>

Original comment by dalewk...@gmail.com on 2 Sep 2009 at 3:01

Attachments:

GoogleCodeExporter commented 9 years ago 
This issue is now fixed in 1.4.0 release. Cheers!

Original comment by charles....@gmail.com on 12 Jan 2010 at 4:14