jenkins rejects report-junit output when no tests are run

[brooksdavis]

In some circumstances a Kuyafile will end up not running any tests. When this happens, the result of running report-junit on the database file is something like:

<?xml version="1.0" encoding="iso-8859-1"?>
<testsuite>
<properties>
<property name="cwd" value="/root"/>
<property name="env.BLOCKSIZE" value="K"/>
<property name="env.EDITOR" value="vi"/>
<property name="env.GROUP" value="wheel"/>
<property name="env.HOME" value="/root"/>
<property name="env.HOST" value="beri1"/>
<property name="env.HOSTTYPE" value="FreeBSD"/>
<property name="env.LOGNAME" value="root"/>
<property name="env.MACHTYPE" value="mips"/>
<property name="env.MAIL" value="/var/mail/root"/>
<property name="env.OSTYPE" value="FreeBSD"/>
<property name="env.PAGER" value="more"/>
<property name="env.PATH" value="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin"/>
<property name="env.PWD" value="/root"/>
<property name="env.SHELL" value="/bin/csh"/>
<property name="env.SHLVL" value="1"/>
<property name="env.TERM" value="dialup"/>
<property name="env.USER" value="root"/>
<property name="env.VENDOR" value="mips"/>
</properties>
</testsuite>

Jenkins treats this output as an error, I believe due to a lack of any <testcase> elements, but I'm not 100% sure. I might be nice if kyua would insert a pseudo successful test like "no_tests_run" in this case. (I've hit this on an older FreeBSD-CURRENT base where I'm running /usr/tests/*/Kyuafile individually and /usr/tests/secure and /usr/tests/cddl run no tests on mips64.)

[jmmv]

Maybe this is a bug in Jenkins though? Per the XSD definitions in https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd , I understand that there is no requirement for a testsuite element to contain any testcase.

We could workaround this in Kyua by emitting a fake entry... but it'd be good to bring this up with the Jenkins developers for clarification first.

[jmmv]

This is what Jenkins complains about when it encounters are report without results:

Recording test results
ERROR: Step ?Publish JUnit test result report? failed: None of the test reports contained any result
Finished: FAILURE

so your theory is correct.

[jmmv]

The JUnit plugin has an option Do not fail the build on empty test results described as:

If checked, the default behavior of failing a build on missing test result files or empty test results is changed to not affect the status of the build. Please note that this setting make it harder to spot misconfigured jobs or build failures where the test tool does not exit with an error code when not producing test report files.

I kinda agree with this rationale. If you expect the run to not execute any tests, you should tick this option. It seems safer to fail explicitly by default to catch misconfiguration issues.

I have a hard time coming up with a general case where an empty report would make sense, so I feel there is nothing to do here. Do you have any use case in mind?

[jmmv]

I'm going to close this per the rationale in the previous comment. If you have further input, please let me know.