aws-samples / dbt-glue

This repository contains the dbt-glue adapter
Apache License 2.0
101 stars 69 forks source link

Remove update_iceberg_ts column from model with merge incremental strategy #452

Closed aiss93 closed 3 weeks ago

aiss93 commented 1 month ago

@moomindani

Currently, when using the merge incremental strategy with the Iceberg file format, the adapter adds the column update_iceberg_ts, which is not used in the rest of the merge process. This column can be removed, and if such a column is needed, it can be added directly in the model.

Having this column hard-coded can be somewhat constraining, as it may break schema comparisons between tables in certain scenarios.

I'll link a PR to fix this behavious