junitxml: include full output in system-out field

[epot]

Unless I am mistaken, junit supports having system-out under testcase, to capture a failing test standard output. Unfortunately, this does not seem to be supported by this tool. Is there any plan to do so? It would allow feeding other products like Datadog CI visibility for instance.

[dnephin]

Hello, failing test output is captured under testcase.failure , you can see an example here: https://github.com/gotestyourself/gotestsum/blob/main/internal/junitxml/testdata/junitxml-report.golden#L60

This seems to work for most CI systems. Do you have an example of a junit.xml file that isn't working?

[epot]

Ah thanks for this example I did not see it. I have exactly the same layout on my side. My problem is that Datadog CI system expects the standard output to be in system-out. What is under failure is completely put under the "error" area there, whereas I would like a "short" error message there and then logs individually searchable there. You can imagine how ugly the screenshot below can get where it's an integration test and there is a lot of output :).

Maybe I misunderstand the junit standard, but I thought that system-out and system-err were intended to have the standard output. Do you think we could have an option or change the gotestsum default behavior? Or should I look for an alternative?

[dnephin]

Unfortunately there is no real official standard. It seems every system does something a little different. There are a few flags to modify the output in the junit.xml to accomodate different systems

      --junitfile-project-name string               name of the project used in the junit.xml file
      --junitfile-testcase-classname field-format   format the testcase classname field as: full, relative, short (default full)
      --junitfile-testsuite-name field-format       format the testsuite name field as: full, relative, short (default full)

I think to accomplish what you describe here we'll need to wait for this Go proposal https://github.com/golang/go/issues/62728 to be accepted and implemented. With that change we could add a flag to send only the OutputType: error output to the failure field, and send the full log to system-out.

It's possible to send everything to system-out now, if you're interested in submitting a PR to do that, but it wouldn't be anything new. It would be the same text as the failure tag.

Issue #118 is also related I guess. It wasn't said explicitly in that issue, but I'm assuming they were expecting the full output in system-out as well.

I'll re-title this issue to clarify the feature request.

[dnephin]

Looking at https://github.com/windyroad/JUnit-Schema/blob/cfa434d4b8e102a8f55b8727b552a0063ee9044e/JUnit.xsd#L150 I'm still a bit confused about how this is supposed to work.

That system-out tag is at the same level as testcase, meaning it's a sibling to the test case, not a child of it. That makes it look like the system-out is relevant to the entire test suite, not a particular testcase.

Does Datadog CI publish the schema they expect?

[dnephin]

I just noticed that https://github.com/testmoapp/junitxml does actually show it as both a sibling and a child. So I guess that's where it comes from.

[rkeithhill-keysight]

We have this problem as well. We use Jenkins and BlueOcean's CI/CD pipeline view. If we use go-junit-report we get stdout displayed in the Tests view:

I haven't found a way to get this stdout into the --junitfile XML file so it shows up in Jenkins/BlueOcean.

[epot]

@dnephin sorry for the latency, I wanted to circle back with a few Datadog folks and do more tests. Actually I was wrong above: whatever is under testcase.failure actually appears as logs in Datadog. It's just a little weird because it also appears in the test failure, which can be pretty ugly when the test logged a lot of lines (hundreds in some cases). In any case I am interested in putting into a child of testcase the system-out of the test. Do you confirm you would be ok if I submit a PR for that?