Display report after run

[nprizal]

Description

Print report at the end of run that includes run statistics (number of tests that are passed/muted/failed), list of muted tests, and list of failed tests.

========== Buildkite Test Engine Report  ==========
+--------+-------------+---+
| Passed | first run   | 2 |
+        +-------------+---+
|        | after retry | 0 |
+--------+-------------+---+
| Muted  | passed      | 0 |
+        +-------------+---+
|        | failed      | 1 |
+--------+-------------+---+
| Failed |             | 1 |
+--------+-------------+---+
|                TOTAL | 4 |
+--------+-------------+---+

+++ Muted Tests:
- Tomato is vegetable (failed)

+++ Failed Tests:
- Banana is red
===================================================

Changes

• RunResult struct is repurposed to keep track of run result, including all individual test results. The struct now contains the logic to calculate the run status, muted test, and failed tests. It also has a function to return the statistics of the run. (previously it only contains failed test and status as a static attribute)
• Each runner is initialized with the runResult struct, and records all test results to that struct after each run.
• Muted tests filter logic in main package's runTestsWithRetry has been removed.
• Report will be printed at the end of run, and run statistics is displayed as ASCII table using tablewriter package.

[nprizal]

One niggle I had is that it now doesn't easily show the total passed and total failed, maybe we could append that to the end?

Yeah I played with it, and couldn't find a nice way of displaying it. Maybe we can print summary first, and breakdown after that?

2 passed, 2 muted, 1 failed

+--------+-------------+---+
| Passed | first run   | 2 |
+        +-------------+---+
|        | after retry | 0 |
+--------+-------------+---+
| Muted  | passed      | 0 |
+        +-------------+---+
|        | failed      | 1 |
+--------+-------------+---+
| Failed |             | 1 |
+--------+-------------+---+
|                TOTAL | 4 |
+--------+-------------+---+

It'll also be interesting to see how this looks with different numbers on the bk/bk test suite, and the possible double up of the 🔥 Test Analytics Failures 🔥 summary. Another reason why the collector/bktec double up is annoying 😅

It can be a bit noisy. Also, the 🔥 Test Analytics Failures 🔥 only available in ruby collector at the moment.

[niceking]

Ooooo yeah I like that summary line!

It can be a bit noisy. Also, the 🔥 Test Analytics Failures 🔥 only available in ruby collector at the moment.

Yeah but that's what we use on bk/bk which is what we'll be recording for the demo so it'll be rather obvious 😛