Ensure a Slack channel is notified when CLI benchmark tests fail
See if the metabase dashboards can turn "red" (or have some other visual indication) when tests are failing
From @t0yv0:
there is a marker I think in the data though that talks to the run having failed; perhaps some sql munging can change the presentation
The details are - there's basically some accidental coupling in the data harvester that is not robust to individual tests failing so it prefers them. not to fail if some are failing which is most of the time. It's not unreasonable if the success/failure status is written to Redshift like i suspect.
We should:
From @t0yv0:
Related: