Montreal-Analytics / dbt-snowflake-utils

Snowflake-specific utility macros for dbt projects.
Apache License 2.0
105 stars 36 forks source link

warehouse_size() macro no longer works #13

Closed AndrewCathcart closed 3 years ago

AndrewCathcart commented 4 years ago

dbt version i'm using: 0.17.2

Simplest form of the macro;

{% macro warehouse_size() %}
{% if execute %}
{% set full_wh = var('snowflake_utils:full_refresh_run_warehouse', none) %}
{% do return(full_wh) %}
{% endif %}
{% endmacro %}

or

{% macro warehouse_size() %}
{% if execute %}
big_warehouse_name
{% endif %}
{% endmacro %}

This doesn't seem to be liking the 'execute' flag now.

AndrewCathcart commented 3 years ago

I'm currently just using my own warehouse_size macro - i just call this at the top of my models (right below the config).

I have two environments, poc and prod, so I use a different bulk_wh name depending on target.name.

My incremental models also have full_refresh = false, (to avoid accidental full refreshes) in their configs, so this is an extra config flag I check.

{% macro warehouse_size() %}

    {#-- if the model strategy is incremental #}
    {% if execute and model.config.materialized == 'incremental' %}

        {% set relation = adapter.get_relation(this.database, this.schema, this.table) %}

        {#-- set bulk_wh depending on target #}
        {% set bulk_wh = var('bulk_wh_prod', none) if target.name == 'prod' else var('bulk_wh_poc', none) %}
        {% set is_refreshable = config.get('full_refresh', true) %} {#-- defaults to true #}

        {% if bulk_wh is not none %}
            {#-- if the table doesn't exist yet #}
            {% if relation is none %}
                {{ dbt_utils.log_info("Initial Run - Using alternative warehouse " ~ bulk_wh | upper) }}
                {% do run_query("use warehouse " ~ bulk_wh) %}

            {#-- if this is a full-refresh run #}
            {% elif flags.FULL_REFRESH and is_refreshable %}
                {{ dbt_utils.log_info("Full Refresh Run - Using alternative warehouse " ~ bulk_wh | upper) }}
                {% do run_query("use warehouse " ~ bulk_wh) %}

            {% endif %}

        {% endif %}

    {% endif %}

{% endmacro %}
MartinGuindon commented 3 years ago

Thanks for sharing @AndrewCathcart. I'll incorporate these changes back to the package. :)

AndrewCathcart commented 3 years ago

@MartinGuindon would you like me to close this issue? Looks like this repo is no longer being maintained