Closed lostmygithubaccount closed 1 year ago
this is a bit complicated -- even this would fail:
def model(dbt, session):
model_name = "orders"
return dbt.ref(model_name)
intermediate step could be more clear error message
We raised better error message to tell folks not do this, but still want to leave this open to revisit and see if other better options we would like to provide in the future
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.
Interested in this, is there a plan to support dynamic values in the future?
Also interested in this
Hi @ChenyuLInx @lostmygithubaccount , I am currently developing on DBT/Databricks and is facing this issue.
Context:
My Databricks catalog is designed in a way where my client companies are all cleanly separated by schemas (schema names are the companies' identifiers and each company has the exact same table structure). This means that I require being able to run DBT source with dynamic source_name
arguments, which references my python vars
.
(so that the DBT run can reference/transform the correct client company's schema in Databricks).
1) I require using python models for complex transformations with recursive calls that is not achievable via Databricks sql DBT models.
2) I require getting/passing dynamic source_name
variables as such:
def model(dbt, session):
dbt.config(materialized="table", unique_key="id")
source_name = dbt.config.get("DATABRICKS_SCHEMA_NAME")
source_table = dbt.source(source_name, "source_table_name")
...
return final_table
Question/Request:
I would like to understand a little more on why this feature is not supported/encouraged.
1) What is the limitation/risk that I am unaware of here?
2) Is this safeguard something that can maybe be bypassed via some configuration?
(e.g. disallow dynamic source_name
argument by default but has a flag in DBT project or profile config that allows for it to be bypassed)
3) Are there any other workarounds that I can do to achieve what I need in python models?
Note: I am suggesting this because this is achievable if I use sql models in the following manner, so it feels a little inconsistent in principle to have a difference in treatment between python/sql models (see working sql model example below):
{% set source_name = var('DATABRICKS_SCHEMA_NAME') %}
with source_table as (
select
*
from
{{ source(source_name, 'source_table_name') }}
),
...
select * from final_table
Do we have an update on this? This feature can add a lot flexbility to many python data processes. E.g. allowing for f-string input can be great enough. Or is there any workaround currently for Python models?
Interested in this, is there a plan to support dynamic values in the future?
Is this a new bug in dbt-core?
Current Behavior
EDIT: see comment below. this does not fail due to iterables/lists/dictionaries, but rather us statically analyzing the code and being unable to indirectly use strings via variables. thus an even simpler reproduction:
Take
debug1.py
anddebug2.py
, built off of jaffle_shop, as simple examples:I would expect both of these examples to work, but they fail with parsing error below
Expected Behavior
above works
Steps To Reproduce
copy code above into those files
Relevant log output
Environment
Which database adapter are you using with dbt?
snowflake
Additional Context
No response