In the incremental.sql file, it was first checking if the relation was iceberg, instead of checking for full refresh.
See athena connector implementation for reference.
We fix it by modifying the order of the condition, we first check if the user has used the full refresh flag (i.e. -f or --full-refresh).
To check if the user used the flag, we use the dbt core macro should_full_refresh() instead of the currently implemented full_refresh_mode variable. Since we don't need the full_refresh_mode variable anymore, we remove this variable from the code.
Once the full refresh condition is True, we check if it is an iceberg materialization. If yes, we first delete the table using glue__drop_relation macro in adapters.sql. Then, we write the table with the iceberg_write function, that will re-create the table from scratch since it does not exist anymore.
TLDR:
We modify the order of condition by first checking if full refresh is enabled instead of if iceberg.
We check if full refresh is enabled with a dbt-core macro instead of custom implementation.
If full refresh and iceberg, then we delete the table.
Resolves #247
Description
Fixing full refresh with iceberg issue.
In the
incremental.sqlfile, it was first checking if the relation was iceberg, instead of checking for full refresh. See athena connector implementation for reference.We fix it by modifying the order of the condition, we first check if the user has used the full refresh flag (i.e.
-for--full-refresh).To check if the user used the flag, we use the dbt core macro should_full_refresh() instead of the currently implemented full_refresh_mode variable. Since we don't need the full_refresh_mode variable anymore, we remove this variable from the code.
Once the full refresh condition is True, we check if it is an iceberg materialization. If yes, we first delete the table using
glue__drop_relationmacro in adapters.sql. Then, we write the table with theiceberg_writefunction, that will re-create the table from scratch since it does not exist anymore.TLDR:
Checklist
CHANGELOG.mdand added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.