Add `json` format support (not via --jsonfile)

[Pawka]

The gotestsum tool currently has --jsonfile flag. But that does not work well when test suite is large and it is desired to pipe json output while tests are still running.

Imagine we have a tool named json2ci to format test runs for CI system (e.g. Buildkite?). With go test this would be simple as:

$ go test ./... -json | json2ci

With gotestsum it is possible to run following code:

$ gotestsum --jsonfile output.json ./... > /dev/null
$ cat output.json | json2ci

But the code above blocks reporting until gotestsum is done. It is a problem if it takes a while to run the test suite. It would be better to have this:

$ gotestsum --format=json ./... | json2ci

Such feature would bring gotestsum closer to go test in terms of drop-in replacement.

---

Current approach is executing gotestsum in the background and parsing content via tail:

echo "" > test.json
(gotestsum --jsonfile=test.json ./...) &
pid=$!
(tail -f --pid=$pid -n +1 test.json) &
wait $pid

---

P.S. I would like to know if there is a need for such feature or if there are other ways how to achieve the goal. Also understand potential blockers. I would be happy to implement the feature if needed.

[dnephin]

Thank you for your interest in gotestsum!

This is an interesting feature request. It should be relatively easy to implement by adding a new formatter here

https://github.com/gotestyourself/gotestsum/blob/f39e7fa79519138e28b5db5427e9c7a64c6eebb6/testjson/format.go#L239-L240

The formatter could simply return string(event.raw), which would be the input JSON.

Before accepting this feature, I'd like to better understand json2ci, and what value you get from gotestsum in this scenario. Generally the benefit of gotestsum in CI is that the stdout format is human readable, and the file ouputs (junit.xml, or jsonfile) are used to integrate into CI.

Does json2ci exist? What does it do? Is the output a format that gotestsum could product itself, removing the need to pipe output to json2ci ? What benefit does gotestsum provide over go test -json in this scenario (when the output is being consumed by another program) ?

[Pawka]

Thanks for reply!

In this example the json2ci is just artificial binary. We have a similar which prepares output for Buildkite (see Managing log output page in their documentation). We could add another gotestsum format for Buildkite but I think streaming JSON to the output would scale better - anyone would be able to implement any format they want on top of JSON.

The main benefit of gotestsum over go test -json - restarted failed tests instead of whole pipeline. This is especially useful in large codebases (such as we have). Basically the main value is from restarts but we ditch any output generated from gotestsum.

Let me know which direction you'd like to go. Thanks!

[Pawka]

I have created a PR (#313) just in case you'll decide this is something what should be added as a feature to the tool.

[dnephin]

This is great, thank you! I'm happy to add the standard-json format, but I'd like to hide it a bit. I'd rather people come to GitHub and open issues (as you have!) to discuss the use case. That way we can improve gotestsum instead of everyone needing to write the formatting themselves.

Just yesterday I was looking at creating a github actions format (#315) that uses a similar syntax to group logs. I think it should be pretty easy to use the same approach for a buildkite format. I'd love to add those formats to avoid needing to pipe to another binary.

Is there anywhere I can see your build output to get a better idea of what sections are collapsed? Or could you describe it in more detail?

While working on the github actions format a few ideas came to mind:

1. A format similar to the current testname format. Each test would be a group, and expanding the group would show the full output of the test. Currently only failed tests have output, but it wouldn't be too difficult to change to all tests having output hidden by the group.
2. A format similar to the current pkgname format. Each package would be a group, and expanding the group would show the full list of tests in that package, or maybe only the failed test output.

Are either of these similar to what you have now?

[dnephin]

Thank you for implementing this feature! I've created #316 for tracking the new formats. I'd love your feedback on these formats! Either on this issue or the new one would be great.

[Pawka]

Posted short answer on #316.