mozilla / bigquery-etl

Bigquery ETL
https://mozilla.github.io/bigquery-etl
Mozilla Public License 2.0
241 stars 98 forks source link

removing uniqueness check from sync models #5739

Closed chelseybeck closed 1 month ago

chelseybeck commented 1 month ago

With Braze syncs now using incremental updates, we expect some duplicate external IDs

Checklist for reviewer:

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

dataops-ci-bot commented 1 month ago

Integration report for "removing uniqueness check from sync models"

sql.diff

Click to expand! ```diff diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_firefox_subscriptions_sync_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_firefox_subscriptions_sync_v1/checks.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_firefox_subscriptions_sync_v1/checks.sql 2024-06-04 22:46:05.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_firefox_subscriptions_sync_v1/checks.sql 2024-06-04 22:46:00.000000000 +0000 @@ -5,6 +5,3 @@ #warn {{ min_row_count(1) }} - -#warn -{{ is_unique(["EXTERNAL_ID"]) }} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_newsletters_sync_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_newsletters_sync_v1/checks.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_newsletters_sync_v1/checks.sql 2024-06-04 22:46:05.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_newsletters_sync_v1/checks.sql 2024-06-04 22:46:00.000000000 +0000 @@ -5,6 +5,3 @@ #warn {{ min_row_count(1) }} - -#warn -{{ is_unique(["EXTERNAL_ID"]) }} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_products_sync_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_products_sync_v1/checks.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_products_sync_v1/checks.sql 2024-06-04 22:46:05.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_products_sync_v1/checks.sql 2024-06-04 22:46:00.000000000 +0000 @@ -5,6 +5,3 @@ #warn {{ min_row_count(1) }} - -#warn -{{ is_unique(["EXTERNAL_ID"]) }} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_users_sync_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_users_sync_v1/checks.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_users_sync_v1/checks.sql 2024-06-04 22:46:05.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_users_sync_v1/checks.sql 2024-06-04 22:46:00.000000000 +0000 @@ -5,6 +5,3 @@ #warn {{ min_row_count(1) }} - -#warn -{{ is_unique(["EXTERNAL_ID"]) }} diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_waitlists_sync_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_waitlists_sync_v1/checks.sql --- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_waitlists_sync_v1/checks.sql 2024-06-04 22:46:05.000000000 +0000 +++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/braze_external/changed_waitlists_sync_v1/checks.sql 2024-06-04 22:46:00.000000000 +0000 @@ -5,6 +5,3 @@ #warn {{ min_row_count(1) }} - -#warn -{{ is_unique(["EXTERNAL_ID"]) }} ```

Link to full diff