snowplow / dbt-snowplow-utils

Snowplow utility functions to be used in conjunction with the snowplow-web dbt package.
Other
13 stars 6 forks source link

Exclude specific entity versions from combine_column_version macro #89

Closed bgarf closed 2 years ago

bgarf commented 2 years ago

Description & motivation

Add the capability to exclude unwanted versions from the combine_column_version macro.

We've encountered a need for this as we have introduced breaking changes to entity versions that cause BigQuery compilation errors when coalescing. For example, a change in field type across versions. This change allows for the avoidance of this by dropping known problematic versions out of the coalesce.

This change makes the assumption that the column's version is always provided as the last 5 characters at the end of the column name (column_X_X_X).

Example of use

Screenshot 2022-09-16 at 15 49 10

To exclude version 1_0_0 from the coalescing:

{%- set all_fields = snowplow_utils.combine_column_versions(
                                relation=ref('snowplow_web_base_events_this_run'),
                                column_prefix='product_v',
                                exclude_versions=['1_0_1'],
                                ) -%}

select
{% for field in all_fields %}
  {{field}} {%- if not loop.last %},{% endif %}
{% endfor %}

# Renders to:
select
  coalesce(product_1_0_2[safe_offset(0)].name, product_1_0_0[safe_offset(0)].name) as name,
  coalesce(product_1_0_2[safe_offset(0)].id, product_1_0_0[safe_offset(0)].id) as id,
  coalesce(product_1_0_2[safe_offset(0)].specs) as specs,
  coalesce(product_1_0_2[safe_offset(0)].specs.power_rating) as specs_power_rating,
  coalesce(product_1_0_2[safe_offset(0)].specs.volume) as specs_volume

Checklist

snowplowcla commented 2 years ago

Thanks for your pull request. Is this your first contribution to a Snowplow open source project? Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://docs.snowplowanalytics.com/docs/contributing/contributor-license-agreement/ to learn more and sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

bgarf commented 2 years ago

Signed the CLA 👍

paulboocock commented 2 years ago

Bots gone walkies. I can confirm the CLA has been signed.