Closed thijsnijhuis-schiphol closed 10 months ago
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.
Describe the bug
The 'spark__get_external_build_plan' macro does not use the 'source' macro to see of the table already exists. The package does use it in 'spark__create_external_table' to create the table. Therefore, when overruling the source object naming in 'source', the package does create the right objects but it fails to correctly identify if it is already there. If I set the 'ext_full_refresh' to true, it will not generate error but it will always drop/create all tables.
Steps to reproduce
Use azure databricks and create create external table simply pointing the location to a storage account path. Add a 'source' macro to your project and make some change to the default naming of the source objects. In our case, we always want our sources to be named: src___ and be created in the same schema as all other tables. I have created this macro tod o so:
Expected results
The macro that check to see if the table exist should use the same naming as the macro that created the table
Actual results
The macro that checks the exists uses a different table name tan the one that creates them when 'source' is used to overrule the name.
Screenshots and log output
n.a.
System information
The contents of your
packages.yml
file:Which database are you using dbt with? databricks/spark
The output of
dbt --version
:The operating system you're using: Windows 11
The output of
python --version
: Python 3.10.8Additional context
I solved this myself by chaing line 7 and 8 of macros \ plugins \ spark \get_external_build_plan.sql from :
into: