Montreal-Analytics / dbt-snowflake-utils

Snowflake-specific utility macros for dbt projects.
Apache License 2.0
105 stars 36 forks source link

fix: create_tag_if_missing for multiple columns #56

Closed Lorenitz closed 8 months ago

Lorenitz commented 9 months ago

During tag creation, if two columns use the same tag, the macro is trying to create the same tag twice.

The error message we get is the one below:

Object 'DB_NAME.SCHEMA_NAME.TAG_NAME' already exists.

In our case, we apply tags using masking policies in Snowflake. And we want to apply the same tag for different columns.

The problem here is in macro create_tag_if_missing:

{% macro create_tag_if_missing(all_tag_names,new_tag) %}
    {{ log('all_tag_names:', info=True) }}
    {{ log(all_tag_names, info=True) }}
    {% if new_tag.split('.')[2] not in all_tag_names %}
        {{ log('Creating missing tag '+new_tag, info=True) }}
        {%- call statement('main', fetch_result=True) -%}
            create tag {{new_tag}}
        {%- endcall -%}
        {{ all_tag_names.append(new_tag)}}
        {{ log(load_result('main').data, info=True) }}
    {% else %}
        {{ log('Tag already exists: '+new_tag, info=True) }}
    {% endif %}
{% endmacro %}

From the code above, when checking the new_tag.split(.)[2], meaning, the tag name, later in all_tag_name.append(tag_name), it should also add the split(.)[2], otherwise it is adding the complete new_tag name, considering as: data_base.schema_name.tag_name. We should append only the tag_name.

After applying the change below, adding as well the split we have:

{% macro create_tag_if_missing(all_tag_names,new_tag) %}
    {{ log('all_tag_names:', info=True) }}
    {{ log(all_tag_names, info=True) }}
    {% if new_tag.split('.')[2] not in all_tag_names %}
        {{ log('Creating missing tag '+new_tag, info=True) }}
        {%- call statement('main', fetch_result=True) -%}
            create tag {{new_tag}}
        {%- endcall -%}
        {{ all_tag_names.append(new_tag.split('.')[2])}}
        {{ log(load_result('main').data, info=True) }}
    {% else %}
        {{ log('Tag already exists: '+new_tag, info=True) }}
    {% endif %}
{% endmacro %}

This fixes the all_tag_names.append

Lorenitz commented 8 months ago

@arjunthangaraj and @JFrackson, could you please have a look to this PR? 🙏 Thank you very much indeed

JFrackson commented 8 months ago

Closing this PR as @calleo 's earlier PR just fixed this. Thanks for both of your work on identifying this and proposing identical fixes!