search

gotestyourself / gotestsum

'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.
Apache License 2.0
2.03k stars 119 forks source link

Add github-actions and buildkite formats #316

Open dnephin opened 1 year ago

dnephin commented 1 year ago

Github actions supports grouping log lines.

Buildkite supports managing log output.

Gitlab CI supports collapsed sections.

Teamcity support: https://github.com/gotestyourself/gotestsum/issues/316#issuecomment-1611446932

These allow some log output to be collapsed (hidden), and expanded by clicking on an arrow. gotestsum could use these features to provide better output when tests on run on these systems.

See https://github.com/gotestyourself/gotestsum/issues/312#issuecomment-1488048968 for some discussion about a buildkite format, and #315 for a first draft of the github actions format.

Some options for these formats:

  1. A format similar to the current testname format. Each test would be a group, and expanding the group would show the full output of the test. Currently only failed tests have output, but it wouldn't be too difficult to change to all tests having output hidden by the group.
  2. A format similar to the current pkgname format. Each package would be a group, and expanding the group would show the full list of tests in that package, or maybe only the failed test output.
dnephin commented 1 year ago

I haven't been able to find similar features in other CI systems, but if anyone knows of similar features please do leave a comment with a link to the docs!

Pawka commented 1 year ago

Is there anywhere I can see your build output to get a better idea of what sections are collapsed? Or could you describe it in more detail?

Answering comment from the previous discussion (since it make more sense to keep context here). I think following rules would work:

  1. Grouping should be made per root test. E.g. subtest should not be logged as separate groups.
  2. Succeeded test gropus are collapsed.
  3. Groups with failed tests are expanded.

However there are some edge cases which should be handled such as timeouts or panics which does not make test as "failed". In such cases code block where failure happened also should be expanded.

vitaliyf commented 1 year ago

TeamCity: https://www.jetbrains.com/help/teamcity/service-messages.html and for example: https://github.com/onsi/ginkgo/blob/master/reporters/teamcity_report.go

phisco commented 3 months ago
  • Grouping should be made per root test. E.g. subtest should not be logged as separate groups.

I arrived here looking exactly for this, it wasn't implemented in the end, right? CC @vitaliyf @dnephin

dnephin commented 3 months ago

At this time there is no way to group by root test, that is correct.

If someone is interested, that sounds like a nice addition as either a new format, or an option to modify the existing github actions format.