Load test observability: automated regression detection framework

[darshan-sj]

Automatic regression detection framework for Dataflow template Load tests.

• Fetch Load test metrics data from a Datasource (Bigquery)
• Run regression detection algorithm on the dataset
• Create a github issue on failure.

PR contents:

• Framework components for automated Load test regression detection
  • BigQueryDatasetFetcher to fetch metrics data from BQ
  • MetricDeviationConditionCheck to detect regression.
• Sample load test observer for DatastreamToSpanner template
• Sets up weekly github workflow to run the Observer along side Load tests for Spanner templates.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 43.19%. Comparing base (3ce80a4) to head (72aa026). Report is 10 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1756 +/- ## ============================================ + Coverage 43.09% 43.19% +0.10% - Complexity 3471 3485 +14 ============================================ Files 829 829 Lines 48418 48418 Branches 5193 5193 ============================================ + Hits 20866 20915 +49 + Misses 25874 25826 -48 + Partials 1678 1677 -1 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.66% <ø> (+0.33%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.90% <ø> (ø)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.11% <ø> (+0.05%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `68.93% <ø> (+1.30%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `84.13% <ø> (+0.03%)` | :arrow_up: | [see 7 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1756/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)

[darshan-sj]

IT failures were unrelated to this change