m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Add verification of unified views #1119

Open stephen-soltesz opened 1 year ago

stephen-soltesz commented 1 year ago

Due to inadequate pre-production verification, the deployment of https://github.com/m-lab/etl/pull/1118 resulted in service disruption for M-Lab BQ users.

The ideal approach is to have the bigquery-exporter bq_daily_discuss.sql query reference the unified views so that we can generate an alert. However, the unified views are very inefficient today, see: m-lab/prometheus-support/issues/894

This change adds a post-schema change, and pre-deployment verification of the project-local unified views.


This change is Reviewable