mozilla / bigquery-etl

Bigquery ETL
https://mozilla.github.io/bigquery-etl
Mozilla Public License 2.0
253 stars 100 forks source link

Remove `external_downstream_tasks` metadata from `unified_metrics_v1` #5734

Closed sean-rose closed 4 months ago

sean-rose commented 4 months ago

The kpi_forecasting DAG doesn't currently have a wait_for_unified_metrics task.


Checklist for reviewer:

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

dataops-ci-bot commented 4 months ago

Integration report for "Remove external_downstream_tasks metadata from unified_metrics_v1."

sql.diff

Click to expand! ```diff diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_unified.py /tmp/workspace/generated-sql/dags/bqetl_unified.py --- /tmp/workspace/main-generated-sql/dags/bqetl_unified.py 2024-06-04 18:40:43.000000000 +0000 +++ /tmp/workspace/generated-sql/dags/bqetl_unified.py 2024-06-04 18:46:30.000000000 +0000 @@ -188,13 +188,6 @@ execution_date="{{ (execution_date - macros.timedelta(seconds=10800)).isoformat() }}", ) - ExternalTaskMarker( - task_id="kpi_forecasting__wait_for_unified_metrics", - external_dag_id="kpi_forecasting", - external_task_id="wait_for_unified_metrics", - execution_date="{{ (execution_date + macros.timedelta(seconds=3600)).isoformat() }}", - ) - checks__fail_telemetry_derived__unified_metrics__v1_external.set_upstream( checks__fail_telemetry_derived__unified_metrics__v1 ) @@ -217,20 +210,6 @@ retries=0, ) - with TaskGroup( - "checks__warn_telemetry_derived__unified_metrics__v1_external", - ) as checks__warn_telemetry_derived__unified_metrics__v1_external: - ExternalTaskMarker( - task_id="kpi_forecasting__wait_for_unified_metrics", - external_dag_id="kpi_forecasting", - external_task_id="wait_for_unified_metrics", - execution_date="{{ (execution_date + macros.timedelta(seconds=3600)).isoformat() }}", - ) - - checks__warn_telemetry_derived__unified_metrics__v1_external.set_upstream( - checks__warn_telemetry_derived__unified_metrics__v1 - ) - telemetry_derived__rolling_cohorts__v1 = bigquery_etl_query( task_id="telemetry_derived__rolling_cohorts__v1", destination_table="rolling_cohorts_v1", @@ -278,20 +257,6 @@ depends_on_past=False, ) - with TaskGroup( - "telemetry_derived__unified_metrics__v1_external", - ) as telemetry_derived__unified_metrics__v1_external: - ExternalTaskMarker( - task_id="kpi_forecasting__wait_for_unified_metrics", - external_dag_id="kpi_forecasting", - external_task_id="wait_for_unified_metrics", - execution_date="{{ (execution_date + macros.timedelta(seconds=3600)).isoformat() }}", - ) - - telemetry_derived__unified_metrics__v1_external.set_upstream( - telemetry_derived__unified_metrics__v1 - ) - checks__fail_telemetry_derived__unified_metrics__v1.set_upstream( telemetry_derived__unified_metrics__v1 ) diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/unified_metrics_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/unified_metrics_v1/metadata.yaml --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/unified_metrics_v1/metadata.yaml 2024-06-04 18:38:47.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/unified_metrics_v1/metadata.yaml 2024-06-04 18:43:25.000000000 +0000 @@ -19,10 +19,6 @@ owner1: loines scheduling: dag_name: bqetl_unified - external_downstream_tasks: - - task_id: wait_for_unified_metrics - dag_name: kpi_forecasting - execution_delta: 1h bigquery: time_partitioning: type: day ```

Link to full diff