Add a multi package line format

[zalenskivolt]

Add a couple of formatters that outputs multiple packages per line.

Example output:

% gotestsum --format pkgname-compact --format-with-fails --wall-time \
            --format-compact plain -- -count 1 ./...

 367ms ∅ . ∅ cmd/internal/signalhandlerdriver ∅ contrib/notify ∅ internal/dotwriter ∅ internal/log
 680ms ∅ internal/text ✓ cmd/tool/matrix (399ms) ✓ cmd/tool/slowest (500ms) ✓ internal/aggregate (541ms)
 1.22s ✓ internal/filewatcher (623ms) ✓ internal/junitxml (628ms) ✓ testjson (718ms) ✓ cmd (10.59s)

DONE 196 tests, 2 skipped in 12.406s

% gotestsum --format pkgname-compact --wall-time -- -count 1 ./... # using dotwriter to tabulate all packages

 11.3s ∅ . ✓ cmd (10.733s) ∅ cmd/internal/signalhandlerdriver ✓ cmd/tool/matrix (650ms)
 1.39s ✓ cmd/tool/slowest (776ms) ∅ contrib/notify ✓ internal/aggregate (398ms) ∅ internal/dotwriter
 1.19s ✓ internal/filewatcher (495ms) ✓ internal/junitxml (578ms) ∅ internal/log ∅ internal/text
 1.59s ✓ testjson (772ms)

 196 tests, 2 skipped in 11.261s

With -dots[N], output a short dots summary for each package. Failures always start a new line, with the full package name:

go-coap(add-error) % gotestsum --format pkgname-compact --format-with-fails --format-hide-empty-pkg \
                               --format-compact dots7 -- -short -count 1 ./...

=== RUN   TestMessageSetPath
    message_test.go:17: foo
--- FAIL: TestMessageSetPath (0.00s)

✓ dtls (5.623s)·······[25] ✓ examples/dtls/pki (382ms)· ✓ message (315ms)·······[31]
✓ message/codes (247ms)····· ✓ message/noresponse (241ms)······
✖  message/pool (240ms)✖·[24] ✓ message/status (233ms)· ✓ mux (221ms)········· ✓ net (2.266s)·······[17]
✓ net/blockwise (249ms)·······[45] ✓ net/client/limitParallelRequests (245ms)·······[12]
✓ net/observation (325ms)········· ✓ pkg/cache (233ms)······ ✓ pkg/fn (232ms)· ✓ pkg/sync (236ms)·········
✓ tcp (6.062s)·······[31] ✓ tcp/coder (326ms)·· ✓ udp (8.036s)·······[30] ✓ udp/client (1.983s)·······[44]
✓ udp/coder (311ms)··

 311 tests, 1 failure in 10.818s

A bigger example, with --format-compact short or partial-back, compared to relative (default):

influxdb % gotestsum --format pkgname-compact --format-hide-empty-pkg --format-hivis \
                     --format-compact short -- -count 1 ./...

✅ . (1.374s) ✅ transport (1.536s) ✅ authorization (2.085s) ✅ authorizer (3.78s) ✅ bolt (3.919s)
✅ checks (3.78s) ✅ build_tsi (3.98s) ✅ delete_tsm (2.767s) ✅ dump_tsi (2.516s) ✅ dump_tsm (2.516s)
✅ dump_wal (2.679s) ✅ export_lp (1.529s) ✅ aggregators (1.281s) ✅ report_tsi (1.599s) ✅ report_tsm (1.889s)
✅ type_conflicts (1.98s) ✅ verify_seriesfile (2.454s) ✅ verify_tombstone (2.183s) ✅ verify_tsm (2.095s)
✅ verify_wal (2.11s) ✅ launcher (47.195s) ✅ auth (2.455s) ✅ organization (2.165s) ✅ user (2.15s)
✅ upgrade (8.119s) ✅ context (1.863s) ✅ dashboards (2.796s) ✅ transport (2.953s) ✅ dbrp (3.25s)
✅ gather (2.406s) ✅ http (2.988s) ✅ legacy (2.594s) ✅ influxql (1.196s) ✅ query (1.728s) ✅ gota (820ms)
✅ neldermead (865ms) ✅ v1tests (37.714s) ✅ v1validation (5.496s) ✅ inmem (5.525s) ✅ jsonweb (1.153s)
✅ check (1.15s) ✅ cli (1.533s) ✅ feature (1.166s) ✅ override (1.119s) ✅ io (344ms) ✅ platform (1.704s)
✅ errors (598ms) ✅ prom (818ms) ✅ promtest (1.194s) ✅ signals (2.175s) ✅ tracing (1.359s) ✅ http (1.452s)
✅ kv (6.47s) ✅ migration (2.265s) ✅ all (2.246s) ✅ label (5.444s) ✅ mock (2.364s) ✅ models (3.568s)
✅ notebooks (2.653s) ✅ transport (2.642s) ✅ notification (1.645s) ✅ check (1.69s) ✅ endpoint (2.021s)
✅ service (1.972s) ✅ rule (1.737s) ✅ service (5.42s) ✅ bloom (30.454s) ✅ bytesutil (978ms) ✅ gen (2.872s)
✅ durablequeue (10.822s) ✅ simple8b (1.337s) ✅ escape (1.169s) ✅ hll (1.98s) ✅ fs (898ms) ✅ httpc (3.741s)
✅ jsonnet (754ms) ✅ jsonparser (3.352s) ✅ limiter (6.965s) ✅ metrics (2.754s) ✅ mincore (4.748s)
✅ mmap (657ms) ✅ pool (722ms) ✅ radix (665ms) ✅ rhh (5.252s) ✅ slices (593ms) ✅ snowflake (393ms)
✅ testttp (2.515s) ✅ fields (1.631s) ✅ labels (1.484s) ✅ pkger (2.344s) ✅ predicate (1.643s)
✅ prometheus (1.194s) ✅ query (1.41s) ✅ control (6.394s) ✅ influxdb (3.099s) ✅ rand (1.486s)
✅ remotes (2.328s) ✅ transport (2.338s) ✅ replications (1.697s) ✅ internal (2.448s) ✅ remotewrite (1.499s)
✅ transport (1.294s) ✅ secret (1.644s) ✅ session (1.54s) ✅ snowflake (613ms) ✅ sqlite (998ms)
✅ static (1.059s) ✅ storage (1.452s) ✅ flux (6.68s) ✅ reads (1.768s) ✅ backend (2.865s)
✅ coordinator (1.971s) ✅ executor (3.625s) ✅ middleware (2.487s) ✅ scheduler (16.909s) ✅ options (2.363s)
✅ taskmodel (2.268s) ✅ plugins (802ms) ✅ inputs (852ms) ✅ outputs (985ms) ✅ service (1.605s)
✅ telemetry (1.56s) ✅ tenant (7.338s) ✅ toml (686ms) ✅ tmpl (663ms) ✅ tsdb (2m15.132s) ✅ cursors (922ms)
✅ tsm1 (44.573s) ✅ tsi1 (46.858s) ✅ authorization (3.049s) ✅ coordinator (3.412s) ✅ meta (2.769s)
✅ precreator (1.951s) ✅ retention (2.995s) ✅ storage (1.664s)

 8371 tests, 57 skipped in 224.021s

influxdb % gotestsum --format pkgname-compact --format-hide-empty-pkg --format-hivis \
                     --format-compact partial-back -- -count 1 ./...

✅ . (1.321s) ✅ annotations/transport (1.48s) ✅ authorization (1.984s) ✅ authorizer (3.562s)
✅ bolt (4.051s) ✅ checks (3.775s) ✅ cmd/influxd/inspect/build_tsi (3.393s) ✅ delete_tsm (1.923s)
✅ dump_tsi (1.979s) ✅ dump_tsm (2.347s) ✅ dump_wal (2.221s) ✅ export_lp (1.607s)
✅ report_db/aggregators (1.258s) ✅ ↶report_tsi (1.686s) ✅ report_tsm (1.855s) ✅ type_conflicts (1.938s)
✅ verify_seriesfile (2.454s) ✅ verify_tombstone (2.119s) ✅ verify_tsm (1.889s) ✅ verify_wal (2.147s)
✅ ↶launcher (47.02s) ✅ recovery/auth (2.436s) ✅ organization (1.91s) ✅ user (2.362s) ✅ ↶upgrade (8.138s)
✅ context (1.906s) ✅ dashboards (2.798s) ✅ dashboards/transport (2.971s) ✅ dbrp (3.346s) ✅ gather (2.387s)
✅ http (3.176s) ✅ http/legacy (2.68s) ✅ influxql (1.181s) ✅ influxql/query (1.944s)
✅ query/internal/gota (761ms) ✅ ↶neldermead (880ms) ✅ ↶v1tests (37.564s) ✅ v1validation (5.509s)
✅ inmem (5.902s) ✅ jsonweb (1.349s) ✅ kit/check (1.35s) ✅ cli (1.457s) ✅ feature (1.324s)
✅ feature/override (1.309s) ✅ ↶io (350ms) ✅ platform (1.43s) ✅ platform/errors (680ms) ✅ ↶prom (1.167s)
✅ prom/promtest (1.102s) ✅ ↶signals (2.136s) ✅ tracing (1.361s) ✅ transport/http (1.524s) ✅ kv (6.995s)
✅ kv/migration (2.164s) ✅ migration/all (2.142s) ✅ label (5.143s) ✅ mock (2.411s) ✅ models (3.56s)
✅ notebooks (2.427s) ✅ notebooks/transport (3.374s) ✅ notification (1.721s) ✅ notification/check (2.172s)
✅ endpoint (2.229s) ✅ endpoint/service (1.96s) ✅ ↶rule (1.831s) ✅ rule/service (4.652s)
✅ pkg/bloom (30.06s) ✅ bytesutil (838ms) ✅ data/gen (2.666s) ✅ ↶durablequeue (10.555s)
✅ encoding/simple8b (1.161s) ✅ ↶escape (1.518s) ✅ estimator/hll (2.266s) ✅ ↶fs (1.108s) ✅ httpc (4.175s)
✅ jsonnet (951ms) ✅ jsonparser (1.132s) ✅ limiter (4.849s) ✅ metrics (2.75s) ✅ mincore (4.297s)
✅ mmap (2.076s) ✅ pool (708ms) ✅ radix (708ms) ✅ rhh (5.087s) ✅ slices (571ms) ✅ snowflake (346ms)
✅ testttp (2.573s) ✅ tracing/fields (2.277s) ✅ labels (2.111s) ✅ pkger (2.314s) ✅ predicate (1.43s)
✅ prometheus (1.414s) ✅ query (1.488s) ✅ query/control (5.968s) ✅ stdlib/influxdata/influxdb (2.985s)
✅ rand (1.564s) ✅ remotes (2.458s) ✅ remotes/transport (2.43s) ✅ replications (1.631s)
✅ replications/internal (2.064s) ✅ remotewrite (1.367s) ✅ transport (1.296s) ✅ secret (1.77s)
✅ session (1.766s) ✅ snowflake (756ms) ✅ sqlite (1.07s) ✅ static (993ms) ✅ storage (1.452s)
✅ storage/flux (6.555s) ✅ reads (1.718s) ✅ task/backend (2.955s) ✅ backend/coordinator (2.086s)
✅ executor (3.53s) ✅ middleware (2.803s) ✅ scheduler (17.104s) ✅ ↶options (2.328s) ✅ taskmodel (2.33s)
✅ telegraf/plugins (923ms) ✅ plugins/inputs (719ms) ✅ outputs (970ms) ✅ ↶service (1.764s)
✅ telemetry (1.286s) ✅ tenant (7.814s) ✅ toml (696ms) ✅ tools/tmpl (532ms) ✅ tsdb (2m15.296s)
✅ tsdb/cursors (882ms) ✅ engine/tsm1 (42.915s) ✅ ↶index/tsi1 (42.956s) ✅ v1/authorization (3.722s)
✅ coordinator (3.914s) ✅ services/meta (3.446s) ✅ precreator (1.932s) ✅ retention (3.593s)
✅ storage (2.581s)

 8371 tests, 57 skipped in 223.905s

influxdb % gotestsum --format pkgname-compact --format-hide-empty-pkg --format-hivis \
                     --format-compact relative -- -count 1 ./...

✅ . (1.328s) ✅ annotations/transport (1.692s) ✅ authorization (2.22s) ✅ authorizer (2.985s)
✅ bolt (4.127s) ✅ checks (3.819s) ✅ cmd/influxd/inspect/build_tsi (2.95s)
✅ cmd/influxd/inspect/delete_tsm (1.79s) ✅ cmd/influxd/inspect/dump_tsi (2.025s)
✅ cmd/influxd/inspect/dump_tsm (2.063s) ✅ cmd/influxd/inspect/dump_wal (2.329s)
✅ cmd/influxd/inspect/export_lp (2.237s) ✅ cmd/influxd/inspect/report_db/aggregators (1.262s)
✅ cmd/influxd/inspect/report_tsi (1.704s) ✅ cmd/influxd/inspect/report_tsm (1.705s)
✅ cmd/influxd/inspect/type_conflicts (1.628s) ✅ cmd/influxd/inspect/verify_seriesfile (2.685s)
✅ cmd/influxd/inspect/verify_tombstone (2.041s) ✅ cmd/influxd/inspect/verify_tsm (1.944s)
✅ cmd/influxd/inspect/verify_wal (2.142s) ✅ cmd/influxd/launcher (49.491s)
✅ cmd/influxd/recovery/auth (1.959s) ✅ cmd/influxd/recovery/organization (2.437s)
✅ cmd/influxd/recovery/user (2.403s) ✅ cmd/influxd/upgrade (8.503s) ✅ context (1.579s) ✅ dashboards (3.072s)
✅ dashboards/transport (3.369s) ✅ dbrp (3.67s) ✅ gather (2.33s) ✅ http (3.276s) ✅ http/legacy (2.869s)
✅ influxql (1.325s) ✅ influxql/query (1.667s) ✅ influxql/query/internal/gota (1.125s)
✅ influxql/query/neldermead (1.016s) ✅ influxql/v1tests (37.848s) ✅ influxql/v1validation (6.07s)
✅ inmem (5.54s) ✅ jsonweb (1.132s) ✅ kit/check (1.153s) ✅ kit/cli (1.227s) ✅ kit/feature (1.12s)
✅ kit/feature/override (1.209s) ✅ kit/io (374ms) ✅ kit/platform (2.033s) ✅ kit/platform/errors (1.93s)
✅ kit/prom (2.265s) ✅ kit/prom/promtest (1.956s) ✅ kit/signals (3.042s) ✅ kit/tracing (1.935s)
✅ kit/transport/http (1.535s) ✅ kv (6.151s) ✅ kv/migration (2.742s) ✅ kv/migration/all (2.633s)
✅ label (3.729s) ✅ mock (4.451s) ✅ models (3.633s) ✅ notebooks (2.819s) ✅ notebooks/transport (1.96s)
✅ notification (1.112s) ✅ notification/check (1.706s) ✅ notification/endpoint (1.388s)
✅ notification/endpoint/service (1.984s) ✅ notification/rule (1.627s) ✅ notification/rule/service (4.203s)
✅ pkg/bloom (30.151s) ✅ pkg/bytesutil (1.07s) ✅ pkg/data/gen (4.298s) ✅ pkg/durablequeue (9.668s)
✅ pkg/encoding/simple8b (1.339s) ✅ pkg/escape (1.404s) ✅ pkg/estimator/hll (2.145s) ✅ pkg/fs (885ms)
✅ pkg/httpc (1.479s) ✅ pkg/jsonnet (625ms) ✅ pkg/jsonparser (1.042s) ✅ pkg/limiter (4.805s)
✅ pkg/metrics (960ms) ✅ pkg/mincore (2.395s) ✅ pkg/mmap (315ms) ✅ pkg/pool (304ms) ✅ pkg/radix (530ms)
✅ pkg/rhh (7.296s) ✅ pkg/slices (3.038s) ✅ pkg/snowflake (1.936s) ✅ pkg/testttp (2.551s)
✅ pkg/tracing/fields (866ms) ✅ pkg/tracing/labels (626ms) ✅ pkger (3.249s) ✅ predicate (2.641s)
✅ prometheus (1.19s) ✅ query (1.604s) ✅ query/control (6.095s) ✅ query/stdlib/influxdata/influxdb (2.794s)
✅ rand (1.492s) ✅ remotes (2.352s) ✅ remotes/transport (2.341s) ✅ replications (1.821s)
✅ replications/internal (2.167s) ✅ replications/remotewrite (1.329s) ✅ replications/transport (1.565s)
✅ secret (1.672s) ✅ session (1.71s) ✅ snowflake (735ms) ✅ sqlite (1.068s) ✅ static (971ms)
✅ storage (1.445s) ✅ storage/flux (6.439s) ✅ storage/reads (1.712s) ✅ task/backend (2.74s)
✅ task/backend/coordinator (1.964s) ✅ task/backend/executor (3.479s) ✅ task/backend/middleware (2.459s)
✅ task/backend/scheduler (16.841s) ✅ task/options (2.245s) ✅ task/taskmodel (2.214s)
✅ telegraf/plugins (817ms) ✅ telegraf/plugins/inputs (972ms) ✅ telegraf/plugins/outputs (841ms)
✅ telegraf/service (1.58s) ✅ telemetry (1.339s) ✅ tenant (7.41s) ✅ toml (672ms) ✅ tools/tmpl (585ms)
✅ tsdb (2m18.452s) ✅ tsdb/cursors (1.024s) ✅ tsdb/engine/tsm1 (46.038s) ✅ tsdb/index/tsi1 (44.093s)
✅ v1/authorization (3.349s) ✅ v1/coordinator (3.194s) ✅ v1/services/meta (2.755s)
✅ v1/services/precreator (2.752s) ✅ v1/services/retention (3.885s) ✅ v1/services/storage (1.812s)

 8371 tests, 57 skipped in 227.964s

[dnephin]

Thank you for your interest in contributing new gotestsum formats!

Looking at the example output, the two changes from the existing pkgname format seem to be:

1. grouping of packages on a single line
2. printing the elapsed time at the start of the line

Does that sound right?

What's the motivation for the grouping of packages? It might be difficult to predict how to group packages, as many projects use very different package layouts.

[zalenskivolt]

Thank you for your interest in contributing new gotestsum formats!

Looking at the example output, the two changes from the existing pkgname format seem to be:

1. grouping of packages on a single line
2. printing the elapsed time at the start of the line

Does that sound right?

Thanks! Yes. I added versions with and without -time. Maybe a separate command line option would be better to control things like this? Same with -and-test-fails for both this formatter and pkgname.

What's the motivation for the grouping of packages? It might be difficult to predict how to group packages, as many projects use very different package layouts.

It uses a super simple heuristic for this: as long as the next package is another subpackage on the same level, it will combine them (with one additional small special case). I think this might work fairly well for many projects. As soon as there are multiple sub packages on the same level it should pick up many of them. It should always be obvious which package the shortened names refer to, but for clarity it line breaks and outputs the full package name on failure.

My motivation: For the current project I'm working on, this changes the output from some 65+ lines that I have to scroll through, to just one short overview of 15-20 lines.

It struggles a bit when there are many deeply nested packages. Maybe it could be updated to handle this better. Updated now to allow backing up one step, which helps a lot in a lot of cases.

[zalenskivolt]

It struggles a bit when there are many deeply nested packages. Maybe it could be updated to handle this better.

Updated to allow backing up one step with ↶, for example: ✓ pkg/bloom (1.154s) ✓ data/gen (3.949s) ✓ genx (3.949s) ✓ ↶durablequeue (8.301s) ✓ xbytesutil (1.134s)

[dnephin]

I think this is a cool idea!

Instead of ↶ could we remove the trimming of the shared part of the package path? Every package would be printed with it's relative package path, which removes the need to use ↶ and .../. I think that's going to be more obvious to people and shouldn't add too much extra space. It may even reduce the number of lines, because any number of packages can fill the line (instead of using a heuristic). I expect that should simplify the implementation a bit as well. I guess this new format should still use the dotwriter logic to make sure package names aren't wrapped across lines.

Maybe this new format could be called something like pkgname-compact, so that it follows the same naming convention as pkgname, testname ?

I like your idea of using a --format-with-fails flag to enabling printing failures inline for these package formats. That would let us hide the pkgname-and-test-fails format.

Do we need both the time at the start of the line and after the package name? Keeping it only after the package name seems like it works well.

[zalenskivolt]

I think this is a cool idea!

Thanks!

Instead of ↶ could we remove the trimming of the shared part of the package path? Every package would be printed with it's relative package path, which removes the need to use ↶ and .../. I think that's going to be more obvious to people and shouldn't add too much extra space. It may even reduce the number of lines, because any number of packages can fill the line (instead of using a heuristic). I expect that should simplify the implementation a bit as well. I guess this new format should still use the dotwriter logic to make sure package names aren't wrapped across lines.

At first I read this as just showing the last part of the package name, and this gives a spectacularly short output! Failures still get printed with the full name, so I actually think it might be a useful option. But then I realized you meant printing the full (relative) package paths. This adds quite a bit to the output. For influxdb, many lines fit only two packages for example. But I agree it would be the most obvious format to use by default.

I would love to experiment a bit more around this. I'll add it as a hidden option. I'll also try printing only sibling packages but without any extra line breaks, and just skip all ↶ and .../:s in that case.

Maybe this new format could be called something like pkgname-compact, so that it follows the same naming convention as pkgname, testname ?

Updated.

I like your idea of using a --format-with-fails flag to enabling printing failures inline for these package formats. That would let us hide the pkgname-and-test-fails format.

Updated, pkgname-and-test-fails still works but is hidden.

Do we need both the time at the start of the line and after the package name? Keeping it only after the package name seems like it works well.

I wanted this to see the actual wall-clock time of the messages, from exec.Elapsed(). I updated to use a --wall-time flag, and added support in pkgname and testname as well.

[zalenskivolt]

[short names] might be a useful option

I would love to experiment a bit more around this. I'll add it as a hidden option. I'll also try printing only sibling packages but without any extra line breaks, and just skip all ↶ and .../:s in that case.

Added --format-compact with the options relative(default), short, partial, or partial-back (with ↶). No extra line breaks or …/:s, and a little simplified.

I updated the examples at the top. I think controlling the format might be useful for projects of different sizes and package layouts?

--wall-time flag

I wanted this one to interactively see the time progress, since the test times only give total partial times for lots of things that are running in parallel.

Should this be --format-wall-time to be listed with other format options? Do you think it's useful?

called something like pkgname-compact

I think both of the two versions might be useful. The dotwriter one (pkgname-compact2) for terminal usage, where it is ok to rewrite lines. And the plain one (pkgname-compact), for CI usage for example.

Maybe this should also be controlled by some option?

[zalenskivolt]

I think both of the two versions might be useful. The dotwriter one (pkgname-compact2) for terminal usage, where it is ok to rewrite lines. And the plain one (pkgname-compact), for CI usage for example.

Maybe this should also be controlled by some option?

I updated to control this by just adding -plain to the --format-compact.

Also added --format-compact dots for a compact dots-v2 summary for each package:

∅ . ✓ cmd (10.602s)···[62] ✓ cmd/tool/matrix (406ms)···[10] ✓ cmd/tool/slowest (486ms)·· ∅ contrib/notify

[aslafy-z]

Any news on this PR please? Is it ready to review? Thank you!