Enhancement: nicer test failure output

[damonfeldman]

When tests fail, it is not obvious which test fails without looking inside the /build/test-results files individually (or grep via grep failures=.[1-9] ). Also the output is encoded XML rather than readable text.

This is the file structure I see. There is an error inside TEST-Looping.xml but that is not obvious from teh directory or file structure. The failure contents are not an XML structure, but rather an encoding of XML inside XML. It should ideally be nested, or the stack converted to a typical pure-text stack.

<testsuite errors="0" failures="2" hostname="localhost" name="Looping" tests="2" time="0.049026" timestamp=""><testcase classname="loop.sjs" name="loop.sjs" time="0.021005"><failure type="" message="Error running JavaScript request">&lt;error:error xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:t="http://marklogic.com/roxy/test" xmlns:error="http://marklogic.com/xdmp/error" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
  &lt;error:code&gt;JS-JAVASCRIPT&lt;/error:code&gt;
  &lt;error:name/&gt;
  &lt;error:xquery-version&gt;javascript&lt;/error:xquery-version&gt;
  &lt;error:message&gt;Error running JavaScript request&lt;/error:message&gt;
  &lt;error:format-string&gt;JS-JAVASCRIPT: throw new CompileErrorListError(errorList); -- Error running JavaScript request: CompileErrorListError: Compile Errors&lt;/error:format-string&gt;
  &lt;error:retryable&gt;false&l

[daveegrant]

Agree. It is not always easy to see what failed.

[nlaprell]

This would be a real time saver.

[jonesyface]

I've created a couple of pull requests that should hopefully help resolve this issue at https://github.com/marklogic-community/marklogic-unit-test/pull/61 and https://github.com/marklogic-community/ml-gradle/pull/431.

Although it would probably be nicer to have non-encoded XML in the failure stack, the JUnit schema decrees that the content of the node should be text (https://www.ibm.com/support/knowledgecenter/en/SSUFAU_1.0.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html), hence why (I assume) the XML was encoded in the first place.

There is a "format" parameter on the REST endpoint that allows you to reference an XSLT file to transform the result stack to whatever format you want - it needs to be placed in the marklogic-unit-test-modules/src/main/ml-modules/root/test/formats to you to use it. Perhaps an enhancement to the mlUnitTest task to take a parameter to do the same job would be helpful?

[rjrudin]

I just ran the tests in ml-gradle's unit-test-project, and the exact tests that fail are printed, you may just need to scroll up a little bit depending on how tall your terminal window is.

Here's what's logged:

> Task :mlUnitTest FAILED
Constructing DatabaseClient that will connect to port: 8135
Run teardown scripts: true
Run suite teardown scripts: true
Run code coverage: false
Running all suites...
Done running all suites; time: 37ms

example-failing-tests.xqy > example-failing-tests.xqy FAILED
    Assert Equal failed

2 tests completed, 1 failed

1 test result files were written to: build/test-results/marklogic-unit-test

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':mlUnitTest'.
> There were failing tests. See the test results at: build/test-results/marklogic-unit-test/failed

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
1 actionable task: 1 executed

In addition, the way that the XML files are generated is intended to match what JUnit does. So I don't think it's a good idea to change that format or the directory structure.

[jamesagardner]

I agree with Rob's note about following the gradle defaults for storing test results on a filesystem, and retaining the very common XML structure to allow better integration with other tools.

closing this issue

[jonesyface]

As mentioned in the PR, the proposed change here does not change the JUnit XML structure, it simply replaces the unreadable escaped XML structure contained within the <failure> section of the JUnit document with text that is easier to read. According to the JUnit spec, this part of the document has to be text rather than XML, hence the escaping in the first place. The overall structure of the document remains JUnit XML.

@rjrudin Although you are told in the terminal window which tests have failed, it doesn't tell you any details about why it's failed. This detail is held within the error stack output in the <failure> section of the JUnit XML structure, so it is important that this is readable in order to be able to easily debug the issue.

[jamesagardner]

@jonesyface is correct, this is to deal with making the escaped XML MarkLogic error easier to read and understand. Re-opening both the issue and PR #61 .