aws-samples / dbt-glue

This repository contains the dbt-glue adapter
Apache License 2.0
101 stars 69 forks source link

Unable to rebuild a Delta model materialized as table #419

Closed Jeremynadal33 closed 2 months ago

Jeremynadal33 commented 3 months ago

Describe the bug

When using Delta file format, we can run a table model a first time but when trying to rebuild it.

Steps To Reproduce

Create a simple model materialized as table test_table for instance :

{{
    config(
        materialized='table',
        file_format='delta',
    )
}}

with incoming_data as (
    select 1 as id
)

select * from incoming_data

Run this command a first time (no problem) :

dbt run -s test_table

Run it a second time and you will get the following error :

AnalysisException: Cannot create table ('`xxx`.`test_table`'). The associated location ('s3://xxx/xxx/test_table') is not empty but it's not a Delta table

Expected behavior

We should be able to rebuild a table model at any time without issue.

Screenshots and log output

See Steps To Reproduce

System information

The output of dbt --version:

Core:
  - installed: 1.8.3
  - latest:    1.8.4 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - glue:  1.8.1 - Up to date!
  - spark: 1.8.0 - Up to date!

The operating system you're using: MacBook Pro 2,3 GHz Intel Core i9 8 cœurs The output of python --version:

Python 3.12.4

Ideas to solve the issue

I think there are multiple possibilities to achieve this :