dbt-labs / redshift

Redshift package for dbt (getdbt.com)
https://hub.getdbt.com/dbt-labs/redshift/latest/
Apache License 2.0
99 stars 41 forks source link

Fix redshift_maintenance not working with ref #53

Closed jeremyyeo closed 2 years ago

jeremyyeo commented 2 years ago

Description & motivation

See issue #52 for background.

I tested both mechanism of running redshift_maintenance:

  1. Specifying inclusions via a vacuum_tables_sql() macro:
{% macro vacuumable_tables_sql() %}
select 'analytics' as table_database
       , table_schema
       , table_name
  from analytics.information_schema.tables
 where table_type = 'BASE TABLE'
   and table_schema = 'dbt_jyeo'
{% endmacro %}
~ dbt run-operation redshift_maintenance

23:00:55  Running with dbt=1.0.1
23:00:58  12:00:58 + 1 of 1 Vacuuming "analytics"."dbt_jyeo"."my_table"
23:01:02  12:01:02 + 1 of 1 Analyzing "analytics"."dbt_jyeo"."my_table"
23:01:02  12:01:02 + 1 of 1 Finished "analytics"."dbt_jyeo"."my_table" in 3.43s
  1. Specify exclusions via runtime exclude_schemas and exclude_schemas_like args:
~ dbt run-operation redshift_maintenance --args '{exclude_schemas: ["bloom", "close", "incremental_explore", "spectrum", "trello"], exclude_schemas_like: ["dbt%", "sinter%", "fivetran%", "fishtown%", "google%", "snapshots%", "snowplow%", "github%", "integration%", "my%", "public%", "test%", "quickbooks%", "lever%", "slack%", "drew%", "erin%"]}'

23:06:42  Running with dbt=1.0.1
23:06:46  12:06:46 + 1 of 7 Vacuuming "analytics"."analytics"."fivetran_snowplow_unload"
23:06:46  12:06:46 + 1 of 7 Analyzing "analytics"."analytics"."fivetran_snowplow_unload"
23:06:46  12:06:46 + 1 of 7 Finished "analytics"."analytics"."fivetran_snowplow_unload" in 0.59s
23:06:47  12:06:47 + 2 of 7 Vacuuming "analytics"."analytics"."my_table"
23:06:47  12:06:47 + 2 of 7 Analyzing "analytics"."analytics"."my_table"
23:06:47  12:06:47 + 2 of 7 Finished "analytics"."analytics"."my_table" in 0.58s
23:06:48  12:06:48 + 3 of 7 Vacuuming "analytics"."analytics"."stg_customer_id_map"
23:06:48  12:06:48 + 3 of 7 Analyzing "analytics"."analytics"."stg_customer_id_map"
23:06:48  12:06:48 + 3 of 7 Finished "analytics"."analytics"."stg_customer_id_map" in 0.61s
23:06:49  12:06:49 + 4 of 7 Vacuuming "analytics"."analytics"."stg_dbt_users"
23:06:49  12:06:49 + 4 of 7 Analyzing "analytics"."analytics"."stg_dbt_users"
23:06:49  12:06:49 + 4 of 7 Finished "analytics"."analytics"."stg_dbt_users" in 0.57s
23:06:50  12:06:50 + 5 of 7 Vacuuming "analytics"."analytics"."stg_server_farm_cities"
23:06:50  12:06:50 + 5 of 7 Analyzing "analytics"."analytics"."stg_server_farm_cities"
23:06:51  12:06:51 + 5 of 7 Finished "analytics"."analytics"."stg_server_farm_cities" in 0.57s
23:06:51  12:06:51 + 6 of 7 Vacuuming "analytics"."analytics"."stg_trello_members"
23:06:51  12:06:51 + 6 of 7 Analyzing "analytics"."analytics"."stg_trello_members"
23:06:52  12:06:52 + 6 of 7 Finished "analytics"."analytics"."stg_trello_members" in 0.61s
23:06:52  12:06:52 + 7 of 7 Vacuuming "analytics"."analytics"."warehouse_operation"
23:06:56  12:06:56 + 7 of 7 Analyzing "analytics"."analytics"."warehouse_operation"
23:06:56  12:06:56 + 7 of 7 Finished "analytics"."analytics"."warehouse_operation" in 4.05s

For dbt Labs reviewer:

  1. Put this in your packages.yml and then dbt deps:

    packages:
    - git: https://github.com/dbt-labs/redshift.git
    revision: fix/maintenance-macro-with-ref
  2. Add ref() macro override doing nothing:

    
    {% macro ref(model_name) %}
    
    {% set rel = builtins.ref(model_name) %}
    {% set newrel = rel %}
    {% do return(newrel) %}

{% endmacro %}



3. Try a vacuum by copying the above `dbt run-operation redshift_maintenance --args ...` example.

## Checklist
- [x] I have verified that these changes work locally
- [ ] I have updated the README.md (if applicable)
- [ ] I have added tests & descriptions to my models (and macros if applicable)