search

aws-samples / dbt-glue

This repository contains the dbt-glue adapter
Apache License 2.0
101 stars 69 forks source link

Iceberg table materialization does not work #443

Open sanga8 opened 2 months ago

sanga8 commented 2 months ago

Describe the bug

Iceberg table materialization does not work

Steps To Reproduce

    config(
        materialized='table',
        file_format= 'iceberg'
    )

Produces error:

IllegalArgumentException:` Can not create a Path from an empty `string`

System information

The output of dbt --version:


Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - glue:   1.8.1 - Up to date!
  - spark:  1.8.0 - Up to date!```
sanga8 commented 2 months ago

As an alternative we can use 

    config(
        materialized='incremental',
        incremental_strategy='insert_overwrite',
        file_format= 'iceberg',
    )

As stated in the documentation and code if no partition_by is specified with insert_overwrite strategy then it will overwrite the entire table with new data. Thus it behaves as a table materialization.