Have more control on the file names and attributes when testOutput is used

[GoogleCodeExporter]

At the moment when a test is run with option --testOutput it produces an XML 
file for each test case in the following format:

file name: TEST-Firefox_3613_Windows.LeftPanel.xml
with XML data in the format:
<testsuite name="Firefox_3613_Windows.Test24" errors="0" failures="1" tests="1" 
time="0.0010">

There are some challenges with this approach especially if the results are 
processed later.
1. The name of the files is not stable and predictable and they depend on the 
version of the browser used to run the test. A simple browser upgrade can break 
the work flow.
2. The name of the of the test suite XML attribute is not stable and 
predictable.

I think it will be very beneficial if we can add an alternative way to define 
name of the files and attributes described bellow:

1. Add a symbolic name for each browser opened by the server:
--browser firefox:"C:/Program Files (x86)/Mozilla 
Firefox/firefox.exe",safari:"C:/Program Files 
(x86)/Safari/Safari.exe",ie:"C:/Program Files (x86)/Internet 
Explorer/iexplore.exe",chrome:"C:/Users/twhite/AppData/Local/Google/Chrome/Appli
cation/chrome.exe",opera:"C:/Program Files (x86)/Opera/opera.exe"
Use these names later in all logs and XML files.

2. Define the depth of the browser version recorded in two parts:
    a. a major version number  --browserMinorVersion [0,1].  At the end if the day we are interested if the code works on IE6 or IE8.
    b. an optional minor version where length included in the XML file name is controlled by a parameter --browserMinorVersion [0,1,2,3]

3. For the test suite XML name attribute:
   a. the name should include the name of the test suite only. All other parts could be presented as additional attributes
      <testsuite id="LeftPanel" browser="firefox" browserMajor="3" browserMinor="613" os="windows"   name="Firefox_3613_Windows.Test24" errors="0" failures="1" tests="1" time="0.0010">

4. All data in XML files is saved as attributes and this leads not only to a 
very ugly escaping of all XHTML tags but it can not be processed as XML 
anymore. When XHTML is saved as elements then XML diff tools can be used to 
find out what is different between the expected and actual value.
My proposal is the following format:
<failure>
   <message/>
   <description/>
   <expected/>
   <actual/>
   <error/> 
</failure>

XML files could have the following name structure (using the symbolic names 
defined in point 1) run with the following parameters:
 --browserMinorVersion 1  --browserMinorVersion 0

TEST-firefox_3_windows.LeftPanel.xml
TEST-ie_8_windows.LeftPanel.xml

<testsuite id="LeftPanel" browser="firefox" browserMajor="3" browserMinor="613" 
os="windows"   name="Firefox_3613_Windows.Test24" errors="0" failures="1" 
tests="1" time="0.0010">

Benefits: 
1. File names and attributes are predictable and stable and standard XML 
transformation tools like XSLT or XQuery can be used easily to process the test 
results as part of a work flow.
2. These new naming features do not effect the existing functionality and all 
existing code will work.

Original issue reported on code.google.com by thomas.0007 on 24 Feb 2011 at 4:59

[GoogleCodeExporter]

A few points:

Re: 1. symbolic names
I like the idea but not all browsers are, or can be expected to open off the 
command line.

Re: 2. extra flags  
The command line is pretty messy already, and the ROI seems low on this.

Re: 3, 4
This is the standard junit format. Since it works and is recognized by existing 
parsers, I'm not keen on changing.

All said and done, what you really want is plugin that lets you define the 
output. I'm hoping to finalize the api soon, and add enough documentation to 
make it possible.

Original comment by corbinrs...@gmail.com on 24 Feb 2011 at 5:33