Build in monitoring & alerting

[bbrewington]

Acceptance Criteria:

• (TODO)

[BennyJW]

These are my suggestions:

• Create set of validations for CSV data; with validations for overall data and for each column
• Instantiate Great Expectations (GE) in the pipeline (with Google Cloud Run?) to run the validation set
• Set up trigger of GE when new CSV file uploaded to Cloud Storage
• Load static html results of GE into Cloud Storage (public bucket) for review on Google App Spot
• If validation fails, trigger new scrape (three retries?)
• If validation fails after last retry, alert team members (on Slack notification channel)
• If validation successful, trigger load of validated CSV data into BigQuery