When running an upgrade, users can add a --include-dbt flag to run dbt build (which create the models and runs a test) on tables that exists downstream in the dbt project.
User-visible Changes
Internal Changes
Links
Testing
(aws:data-dev, prefix:development) # arthur.py upgrade augur_outputs.results_next --include-dbt
...
2022-05-31 21:01:58 - INFO - Starting to upgrade 1 relation(s) in 1 schema(s)
2022-05-31 21:01:58 - INFO - Connecting to: host=polaris.dev.harrys.systems port=5439 dbname=development user=etl password=***
...
2022-05-31 21:02:08 - INFO - Finished upgrade step for 'augur_outputs.results_next' (7.48s)
2022-05-31 21:02:08 - INFO - Wrapping up work in 6 worker(s): 1 done, 0 not done (0 cancelled) (7.82s)
2022-05-31 21:02:08 - INFO - Finished with 1 relation(s) in source schemas (7.82s)
2022-05-31 21:02:08 - INFO - None of the selected relations are in transformation schemas
2022-05-31 21:02:10 - INFO - Building DBT image
2022-05-31 21:02:25 - INFO - Executing inside dbt container arthur_dbt:latest: $ dbt build -t dev -s ods-full_funnel_metrics+
2022-05-31 21:02:29 - INFO - arthur_dbt:latest # 21:02:29 Running with dbt=1.0.4
2022-05-31 21:02:29 - INFO - arthur_dbt:latest # 21:02:29 Unable to do partial parsing because config vars, config profile, or config target have changed
2022-05-31 21:02:40 - INFO - arthur_dbt:latest # 21:02:40 Found 152 models, 763 tests, 0 snapshots, 0 analyses, 441 macros, 1 operation, 0 seed files, 0 sources, 0 exposures, 0 metrics
2022-05-31 21:02:40 - INFO - arthur_dbt:latest # 21:02:40
2022-05-31 21:02:43 - INFO - arthur_dbt:latest # 21:02:43 Concurrency: 4 threads (target='dev')
2022-05-31 21:02:43 - INFO - arthur_dbt:latest # 21:02:43
2022-05-31 21:02:43 - INFO - arthur_dbt:latest # 21:02:43 1 of 24 START table model ods.full_funnel_metrics............................... [RUN]
2022-05-31 21:02:49 - INFO - arthur_dbt:latest # 21:02:49 1 of 24 OK created table model ods.full_funnel_metrics.......................... [SELECT in 6.47s]
2022-05-31 21:02:49 - INFO - arthur_dbt:latest # 21:02:49 4 of 24 START test not_null_ods-full_funnel_metrics_branded_sem_distributed_conversions [RUN]
2022-05-31 21:02:49 - INFO - arthur_dbt:latest # 21:02:49 2 of 24 START test not_null_ods-full_funnel_metrics_brand_code.................. [RUN]
2022-05-31 21:02:49 - INFO - arthur_dbt:latest # 21:02:49 3 of 24 START test not_null_ods-full_funnel_metrics_brand_sk.................... [RUN]
2022-05-31 21:02:49 - INFO - arthur_dbt:latest # 21:02:49 5 of 24 START test not_null_ods-full_funnel_metrics_clicks...................... [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 2 of 24 PASS not_null_ods-full_funnel_metrics_brand_code........................ [PASS in 0.43s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 6 of 24 START test not_null_ods-full_funnel_metrics_daily_attributed_branded_sem [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 4 of 24 PASS not_null_ods-full_funnel_metrics_branded_sem_distributed_conversions [PASS in 0.49s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 7 of 24 START test not_null_ods-full_funnel_metrics_distributed_brand_spend_local [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 3 of 24 PASS not_null_ods-full_funnel_metrics_brand_sk.......................... [PASS in 0.50s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 5 of 24 PASS not_null_ods-full_funnel_metrics_clicks............................ [PASS in 0.51s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 8 of 24 START test not_null_ods-full_funnel_metrics_distributed_brand_spend_usd. [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 9 of 24 START test not_null_ods-full_funnel_metrics_distributed_conversions..... [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 6 of 24 PASS not_null_ods-full_funnel_metrics_daily_attributed_branded_sem...... [PASS in 0.39s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 10 of 24 START test not_null_ods-full_funnel_metrics_dw__loaded_at.............. [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 7 of 24 PASS not_null_ods-full_funnel_metrics_distributed_brand_spend_local..... [PASS in 0.38s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 11 of 24 START test not_null_ods-full_funnel_metrics_impressions................ [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 9 of 24 PASS not_null_ods-full_funnel_metrics_distributed_conversions........... [PASS in 0.41s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 12 of 24 START test not_null_ods-full_funnel_metrics_lowest_level_source........ [RUN]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 8 of 24 PASS not_null_ods-full_funnel_metrics_distributed_brand_spend_usd....... [PASS in 0.44s]
2022-05-31 21:02:50 - INFO - arthur_dbt:latest # 21:02:50 13 of 24 START test not_null_ods-full_funnel_metrics_metric_date................ [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 10 of 24 PASS not_null_ods-full_funnel_metrics_dw__loaded_at.................... [PASS in 0.39s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 14 of 24 START test not_null_ods-full_funnel_metrics_metric_month............... [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 11 of 24 PASS not_null_ods-full_funnel_metrics_impressions...................... [PASS in 0.41s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 15 of 24 START test not_null_ods-full_funnel_metrics_metric_week................ [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 13 of 24 PASS not_null_ods-full_funnel_metrics_metric_date...................... [PASS in 0.36s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 16 of 24 START test not_null_ods-full_funnel_metrics_region_code................ [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 12 of 24 PASS not_null_ods-full_funnel_metrics_lowest_level_source.............. [PASS in 0.40s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 17 of 24 START test not_null_ods-full_funnel_metrics_region_name................ [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 14 of 24 PASS not_null_ods-full_funnel_metrics_metric_month..................... [PASS in 0.35s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 18 of 24 START test not_null_ods-full_funnel_metrics_total_attributed_spend_local [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 15 of 24 PASS not_null_ods-full_funnel_metrics_metric_week...................... [PASS in 0.41s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 19 of 24 START test not_null_ods-full_funnel_metrics_total_attributed_spend_usd. [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 16 of 24 PASS not_null_ods-full_funnel_metrics_region_code...................... [PASS in 0.38s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 20 of 24 START test not_null_ods-full_funnel_metrics_total_brand_spend_local.... [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 17 of 24 PASS not_null_ods-full_funnel_metrics_region_name...................... [PASS in 0.40s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 21 of 24 START test not_null_ods-full_funnel_metrics_total_brand_spend_usd...... [RUN]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 18 of 24 PASS not_null_ods-full_funnel_metrics_total_attributed_spend_local..... [PASS in 0.35s]
2022-05-31 21:02:51 - INFO - arthur_dbt:latest # 21:02:51 22 of 24 START test not_null_ods-full_funnel_metrics_total_cost_local........... [RUN]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 19 of 24 PASS not_null_ods-full_funnel_metrics_total_attributed_spend_usd....... [PASS in 0.38s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 23 of 24 START test not_null_ods-full_funnel_metrics_total_cost_usd............. [RUN]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 21 of 24 PASS not_null_ods-full_funnel_metrics_total_brand_spend_usd............ [PASS in 0.38s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 24 of 24 START test not_null_ods-full_funnel_metrics_unique_user_count.......... [RUN]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 20 of 24 PASS not_null_ods-full_funnel_metrics_total_brand_spend_local.......... [PASS in 0.43s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 22 of 24 PASS not_null_ods-full_funnel_metrics_total_cost_local................. [PASS in 0.60s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 23 of 24 PASS not_null_ods-full_funnel_metrics_total_cost_usd................... [PASS in 0.46s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 24 of 24 PASS not_null_ods-full_funnel_metrics_unique_user_count................ [PASS in 0.46s]
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Running 1 on-run-end hook
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Granting Schema and Table Access in this schema: ods to these group(s): ['etl_ro', 'analyst_ro', 'analyst_v2_ro', 'api_ro']
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Granting Schema and Table Access in this schema: ods to this group: etl_ro
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Granting Schema and Table Access in this schema: ods to this group: analyst_ro
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Granting Schema and Table Access in this schema: ods to this group: analyst_v2_ro
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 Granting Schema and Table Access in this schema: ods to this group: api_ro
2022-05-31 21:02:52 - INFO - arthur_dbt:latest # 21:02:52 1 of 1 START hook: harrys_data_warehouse.on-run-end.0........................... [RUN]
2022-05-31 21:02:53 - INFO - arthur_dbt:latest # 21:02:53 1 of 1 OK hook: harrys_data_warehouse.on-run-end.0.............................. [ALTER DEFAULT PRIVILEGES in 0.73s]
2022-05-31 21:02:53 - INFO - arthur_dbt:latest # 21:02:53
2022-05-31 21:02:53 - INFO - arthur_dbt:latest # 21:02:53
2022-05-31 21:02:53 - INFO - arthur_dbt:latest # 21:02:53 Finished running 1 table model, 23 tests, 1 hook in 12.97s.
2022-05-31 21:02:54 - INFO - arthur_dbt:latest # 21:02:54
2022-05-31 21:02:54 - INFO - arthur_dbt:latest # 21:02:54 Completed successfully
2022-05-31 21:02:54 - INFO - arthur_dbt:latest # 21:02:54
2022-05-31 21:02:54 - INFO - arthur_dbt:latest # 21:02:54 Done. PASS=24 WARN=0 ERROR=0 SKIP=0 TOTAL=24
2022-05-31 21:02:54 - INFO - Ran 'upgrade' for 72.19s and finished successfully!
(aws:data-dev, prefix:youssef) # arthur.py upgrade mock_arthur_schema --include-dbt --with-staging
2022-05-31 21:18:35 - INFO - Starting log for redshift_etl v1.62.0 with ETL ID 3241646BB7884EE3
2022-05-31 21:18:35 - INFO - Command line: "/opt/local/redshift_etl/venv/bin/arthur.py upgrade mock_arthur_schema --with-dbt --with-staging"
2022-05-31 21:18:35 - INFO - Release information: toplevel=/Users/youssefnaim/code/arthur-redshift-etl, commit=1f8f20e261a61b0874c294e6ac170ff64dc9741f (ynaim94/DENG-1806/show-downstream-dbt), date=2022-05-27 13:08:30 -0400, warning=locally modified files exist
2022-05-31 21:18:35 - INFO - Loading settings from '/opt/src/arthur-redshift-etl/python/etl/config/default_settings.yaml'
2022-05-31 21:18:35 - INFO - Loading settings from '/opt/data-warehouse/config_data_development/aws.yaml'
2022-05-31 21:18:35 - INFO - Loading environment variables from '/opt/data-warehouse/config_data_development/credentials.sh'
2022-05-31 21:18:35 - INFO - Loading environment variables from '/opt/data-warehouse/config_data_development/credentials_old.sh'
2022-05-31 21:18:35 - INFO - Loading settings from '/opt/data-warehouse/config_data_development/harrys.yaml'
2022-05-31 21:18:35 - INFO - Loading settings from '/opt/data-warehouse/config_data_development/harrys_dev.yaml'
2022-05-31 21:18:35 - INFO - Starting background server for monitor on port 8086
2022-05-31 21:18:35 - INFO - Found existing events table 'dw-etl-dev-youssef-events' in DynamoDB (status: ACTIVE)
2022-05-31 21:18:36 - INFO - Looking for files in 's3://harrys-data-warehouse-dev/youssef/data/'
2022-05-31 21:18:36 - INFO - Looking for files in 's3://harrys-data-warehouse-dev/youssef/schemas/'
2022-05-31 21:18:36 - INFO - Found 2 matching file(s) for 1 table(s)
2022-05-31 21:18:36 - INFO - Loading table design for 1 relation(s) to mark required relations
2022-05-31 21:18:36 - INFO - Finished loading 1 table design file(s) (0.12s)
2022-05-31 21:18:36 - INFO - Marked 0 relation(s) as required based on selector: ['analytics.crm_user_profile', 'dw.fact*', 'ods.*']
2022-05-31 21:18:36 - INFO - Pondering execution order of 1 relation(s)
2022-05-31 21:18:36 - INFO - Reusing previously computed execution order of 1 relation(s)
2022-05-31 21:18:36 - INFO - Starting to upgrade 1 relation(s) in 1 schema(s)
2022-05-31 21:18:36 - INFO - Connecting to: host=polaris.dev.harrys.systems port=5439 dbname=development user=etl password=***
2022-05-31 21:18:36 - INFO - Creating schema 'etl_staging$mock_arthur_schema'
2022-05-31 21:18:38 - INFO - Granting access in 'etl_staging$mock_arthur_schema' to 'etl_ro'
2022-05-31 21:18:38 - INFO - None of the relations are in source schemas
2022-05-31 21:18:38 - INFO - Connecting to: host=polaris.dev.harrys.systems port=5439 dbname=development user=etl password=***
2022-05-31 21:18:39 - INFO - Using 10 WLM queue slot(s)
2022-05-31 21:18:39 - INFO - Setting timeout for statements running in Redshift to 7200000 ms
2022-05-31 21:18:39 - INFO - Starting upgrade step for 'mock_arthur_schema.table' (1/1)
2022-05-31 21:18:39 - INFO - Dropping table 'mock_arthur_schema.table' (in staging)
2022-05-31 21:18:40 - INFO - Creating table 'mock_arthur_schema.table' (in staging)
2022-05-31 21:18:41 - INFO - Granting select access on 'mock_arthur_schema.table' (in staging) to 'etl_ro'
2022-05-31 21:18:42 - INFO - Inserting data into 'etl_staging$mock_arthur_schema.table' from query
2022-05-31 21:18:43 - INFO - Running analyze step on table 'mock_arthur_schema.table' (in staging)
2022-05-31 21:18:44 - INFO - No constraints to verify for 'mock_arthur_schema.table'
2022-05-31 21:18:44 - INFO - Found 1 row(s) in 'mock_arthur_schema.table' (in staging)
2022-05-31 21:18:44 - INFO - Finished upgrade step for 'mock_arthur_schema.table' (5.07s)
2022-05-31 21:18:44 - INFO - Finished with 1 relation(s) in transformation schemas (5.40s)
2022-05-31 21:18:45 - INFO - Building DBT image
2022-05-31 21:19:02 - INFO - Executing inside dbt container arthur_dbt:latest: $ dbt build -t etl_staging -s required_dummy-table+ required_dummy-table+ required_dummy-table+
2022-05-31 21:19:06 - INFO - arthur_dbt:latest # 21:19:06 Running with dbt=1.0.4
2022-05-31 21:19:06 - INFO - arthur_dbt:latest # 21:19:06 Found 26 models, 0 tests, 0 snapshots, 0 analyses, 440 macros, 1 operation, 0 seed files, 0 sources, 0 exposures, 0 metrics
2022-05-31 21:19:06 - INFO - arthur_dbt:latest # 21:19:06
2022-05-31 21:19:08 - INFO - arthur_dbt:latest # 21:19:08 Concurrency: 4 threads (target='etl_staging')
2022-05-31 21:19:08 - INFO - arthur_dbt:latest # 21:19:08
2022-05-31 21:19:08 - INFO - arthur_dbt:latest # 21:19:08 1 of 1 START table model etl_staging$required_dummy.table....................... [RUN]
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13 1 of 1 OK created table model etl_staging$required_dummy.table.................. [SELECT in 5.52s]
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13 Running 1 on-run-end hook
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13 Granting Schema and Table Access in this schema: etl_staging$required_dummy to these group(s): ['etl_ro', 'api_ro']
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13 Granting Schema and Table Access in this schema: etl_staging$required_dummy to this group: etl_ro
2022-05-31 21:19:13 - INFO - arthur_dbt:latest # 21:19:13 Granting Schema and Table Access in this schema: etl_staging$required_dummy to this group: api_ro
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:13 1 of 1 START hook: test_hdw_dbt.on-run-end.0.................................... [RUN]
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14 1 of 1 OK hook: test_hdw_dbt.on-run-end.0....................................... [ALTER DEFAULT PRIVILEGES in 0.60s]
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14 Finished running 1 table model, 1 hook in 7.69s.
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14 Completed successfully
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14
2022-05-31 21:19:14 - INFO - arthur_dbt:latest # 21:19:14 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2022-05-31 21:19:15 - INFO - Ran 'upgrade' for 39.76s and finished successfully!
Description
When running an upgrade, users can add a
--include-dbtflag to rundbt build(which create the models and runs a test) on tables that exists downstream in the dbt project.User-visible Changes
Internal Changes
Links
Testing
With staging (Using this branch https://github.com/harrystech/harrys-data-warehouse/tree/ynaim94/deng-1314/mock-dw)
Deploy Notes