When using Delta file format, we can run a table model a first time but when trying to rebuild it.
Steps To Reproduce
Create a simple model materialized as table test_table for instance :
{{
config(
materialized='table',
file_format='delta',
)
}}
with incoming_data as (
select 1 as id
)
select * from incoming_data
Run this command a first time (no problem) :
dbt run -s test_table
Run it a second time and you will get the following error :
AnalysisException: Cannot create table ('`xxx`.`test_table`'). The associated location ('s3://xxx/xxx/test_table') is not empty but it's not a Delta table
Expected behavior
We should be able to rebuild a table model at any time without issue.
Screenshots and log output
See Steps To Reproduce
System information
The output of dbt --version:
Core:
- installed: 1.8.3
- latest: 1.8.4 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- glue: 1.8.1 - Up to date!
- spark: 1.8.0 - Up to date!
The operating system you're using:
MacBook Pro 2,3 GHz Intel Core i9 8 cœurs
The output of python --version:
Python 3.12.4
Ideas to solve the issue
I think there are multiple possibilities to achieve this :
The easier : add a new materialization delta_table_replace like the already existing iceberge_table_replace
The cleaner (in my opinion) : remove the glue__create_table_as macro and use a new macro (create_glue_table_as for instance) to use in the whole project. This will allow passing on a new parameter to easily be able to pass a should_replace option that adds the "or replace" when called in the table materialization. I would like to hear if there is another reason for keeping the adapter macro since this one will only be used inside the project ? What do you guys think ?
Describe the bug
When using Delta file format, we can run a table model a first time but when trying to rebuild it.
Steps To Reproduce
Create a simple model materialized as table test_table for instance :
Run this command a first time (no problem) :
Run it a second time and you will get the following error :
Expected behavior
We should be able to rebuild a table model at any time without issue.
Screenshots and log output
See Steps To Reproduce
System information
The output of
dbt --version
:The operating system you're using: MacBook Pro 2,3 GHz Intel Core i9 8 cœurs The output of
python --version
:Ideas to solve the issue
I think there are multiple possibilities to achieve this :