<testsuites> (everything)
<testsuite> (directory of tests)
<testcase> (a test directory containing steps)
Now:
<testsuites> (everything)
<testsuite> (directory of tests)
<testsuite> (a test directory containing steps)
<testcase> (a single step)
See example at the bottom.
Why
Improved automation
The primary driver for this change is increasing quality of bug reports that are filed by automation based on the JUnit report files by junit2jira.
The tests in the stackrox suite are rather heavyweight, because the operands under test are heavyweight - resource-hungry and slow to set up. This means each test consists of a rather long series of steps - each making assertions after flipping a certain feature on or off.
However this means that each test can fail for a variety of mostly unrelated reasons.
One important thing to understand is how junit2jira coalesces reports. It treats the concatenation of <testcase>'s classname and name attributes as the key (title) for tickets. Once its detects a ticket with the same title already exists, then it appends a comment to this existing ticket, rather than create a new one.
However given the above, this leads to ever-growing tickets with just vaguely related failure root causes (after a while).
Changing the report to map each <testcase> to a kuttl step, rather than whole test, is going to make these automated reports more meaningful and easier to deal with.
More detailed time data
An additional advantage is the more granular timing data - shows which steps are the slow ones.
Caveats
While it seems that nested <testsuite> elements are widely accepted as valid ([1], [2] not all JUnit parsing tools might support this out of the box, so there is some potential for breaking existing workflows.
FWIW, so far there were no comments on slack about this change, so perhaps there are not many report users.
If needed, I could add a knob to make this high-granularity opt-in.
What
Change the report schema.
Previously:
Now:
See example at the bottom.
Why
Improved automation
The primary driver for this change is increasing quality of bug reports that are filed by automation based on the JUnit report files by junit2jira.
The tests in the stackrox suite are rather heavyweight, because the operands under test are heavyweight - resource-hungry and slow to set up. This means each test consists of a rather long series of steps - each making assertions after flipping a certain feature on or off.
However this means that each test can fail for a variety of mostly unrelated reasons.
One important thing to understand is how
junit2jiracoalesces reports. It treats the concatenation of<testcase>'sclassnameandnameattributes as the key (title) for tickets. Once its detects a ticket with the same title already exists, then it appends a comment to this existing ticket, rather than create a new one.However given the above, this leads to ever-growing tickets with just vaguely related failure root causes (after a while).
Changing the report to map each
<testcase>to a kuttl step, rather than whole test, is going to make these automated reports more meaningful and easier to deal with.More detailed time data
An additional advantage is the more granular timing data - shows which steps are the slow ones.
Caveats
While it seems that nested
<testsuite>elements are widely accepted as valid ([1], [2] not all JUnit parsing tools might support this out of the box, so there is some potential for breaking existing workflows.FWIW, so far there were no comments on slack about this change, so perhaps there are not many report users.
If needed, I could add a knob to make this high-granularity opt-in.
Example
Before:
After: