aws-samples / dbt-glue

This repository contains the dbt-glue adapter
Apache License 2.0
101 stars 69 forks source link

Fix table materialization for Delta models #420

Closed Jeremynadal33 closed 2 months ago

Jeremynadal33 commented 3 months ago

resolves #419 with the first (maybe not the best) idea

Description

I modified the function get_relation to pass the argument is_delta to the relation's creation.
Then, I added a "or replace" in the glue\_createtable_as to be able to rebuild multiple times the same model.

I am pretty sure we could update the is_iceberg and is_hudi arguments in the same way hence render the iceberg_table_replace useless but I did not have time to test it out

Checklist

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

FYI :

Tests that were run locally

Simple model with minimal table strategy (test_table):

{{
    config(
        materialized='table',
        file_format='delta',
    )
}}

with incoming_data as (
    select 1 as id
)

select * from incoming_data

No problem at first run but when running a second time the following command :

dbt run -s test_table

Got the following error :

15:25:56  Glue adapter: Glue returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.8.4", "profile_name": "data_platform_preparation", "target_name": "local", "node_id": "model.data_platform_preparation.test_table"} */

    create table xxx.test_table
    using delta
        LOCATION 's3://xxx/xxx/test_table/'
        as
with incoming_data as (
    select 1 as id
)

select * from incoming_data
  ''', use_arrow=False, location='s3://xxx'), AnalysisException: Cannot create table ('`lgdev_jnadal`.`test_table`'). The associated location ('s3://xxx/xxx/test_table') is not empty but it's not a Delta table

Now I can rebuild the model without problem, change its schema also works.

moomindani commented 2 months ago

Thanks for your contribution.