Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
492 stars 121 forks source link

[BUG] When derived_columns are provided, all source columns are transformed to uppercase #166

Closed anouar-zh closed 1 year ago

anouar-zh commented 1 year ago

Describe the bug As soon as a use derived_columns: containing a couple of derived columns, suddenly all source columns are transformed to uppercase while I expect them to be unchanged.

Environment

dbt version: 1.3.0 dbtvault version: 0.9.0 Database/Platform: databricks

To Reproduce

The following results in source columns as they where: (lowercase)

{{config(materialized='view')}}

{% set yaml_metadata %}
source_model: raw_stores
{% endset %}  

{% set metadata_dict = fromyaml(yaml_metadata) %}

{% set source_model = metadata_dict['source_model'] %}

{{ dbtvault.stage(include_source_columns=true,
                  source_model=source_model,
                  derived_columns=derived_columns,
                  null_columns=null_columns,
                  hashed_columns=hashed_columns,
                  ranked_columns=ranked_columns) }}

The following code results in a view in which all columns are suddenly uppercase:

{{config(materialized='view')}}

{% set yaml_metadata %}
source_model: raw_stores
derived_columns: 
   RECORD_SOURCE: "!SOURCE"
   LOAD_DATE: DNAHeader_date
   EFFECTIVE_FROM: EffectiveDate

{% endset %}  

{% set metadata_dict = fromyaml(yaml_metadata) %}

{% set source_model = metadata_dict['source_model'] %}
{% set derived_columns = metadata_dict['derived_columns'] %}

{{ dbtvault.stage(include_source_columns=true,
                  source_model=source_model,
                  derived_columns=derived_columns,
                  null_columns=null_columns,
                  hashed_columns=hashed_columns,
                  ranked_columns=ranked_columns) }}

Expected behavior I expect the source column names to be unchanged

DVAlexHiggs commented 1 year ago

0.9.1 Released which addresses this issue. Thanks! Please re-open if this continues