Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
511 stars 131 forks source link

[BUG] error when using a dbt macro as logic for derived column in stage macro #203

Closed graciegoheen closed 1 year ago

graciegoheen commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

Let's say I have a derived column that uses logic that I've stored in a dbt macro to make my code more DRY:

derived_columns:
    order_total_dollars: {{ cents_to_dollars(column_name = order_total) }}
{% macro cents_to_dollars(column_name, precision=2) -%}
    ({{ column_name }} / 100)::numeric(16, {{ precision }})
{%- endmacro %}

I get an error when trying to compile my stage model:

Compilation Error in rpc request (from remote system.sql) Invalid columns_str object provided. Must be a string and not null. > in macro default__as_constant (macros/internal/metadata_processing/as_constant.sql) > called by macro as_constant (macros/internal/metadata_processing/as_constant.sql) > called by macro default__derive_columns (macros/staging/derive_columns.sql) > called by macro derive_columns (macros/staging/derive_columns.sql) > called by macro default__stage (macros/staging/stage.sql) > called by macro stage (macros/staging/stage.sql) > called by rpc request (from remote system.sql)

Environment

dbt version: 1.5 automate_dv version: 0.9.6 Database/Platform: snowflake

Expected behavior A clear and concise description of what you expected to happen.

I would like to be able to use a dbt macro for my derived column definition in my stage files to make my code more DRY.

Screenshots If applicable, add screenshots to help explain your problem.

Log files If applicable, provide dbt log files which include the problem.

Additional context Add any other context about the problem here.

graciegoheen commented 1 year ago

Looks like this error was unrelated!

DVAlexHiggs commented 1 year ago

Looks like this error was unrelated!

Ok no worries! I'll be honest, this is not something we've tested and we have considered including support.

If you get it working let us know! It might be awkward, so if we can add proper support then we'd love to. If it "just works" then even better!

graciegoheen commented 1 year ago

It actually "just works"!

I had a separate derived column set to NULL. I had forgotten to add quotes to the NULL ('NULL'), so that was what was actually throwing the error.

DVAlexHiggs commented 1 year ago

It actually "just works"!

I had a separate derived column set to NULL. I had forgotten to add quotes to the NULL ('NULL'), so that was what was actually throwing the error.

Excellent! Thanks for the feedback 😊