search

chrisgebert / open_alex_snapshot

A `dbt` project to analyze data snapshots from OpenAlex
MIT License
8 stars 0 forks source link

Test different incremental load strategies supported by `dbt-duckdb` #2

Open chrisgebert opened 7 months ago

chrisgebert commented 7 months ago

From https://github.com/duckdb/dbt-duckdb:

dbt-duckdb supports the delete+insert and append strategies for incremental table models, but unfortunately it does not yet support incremental materialization strategies for external models.

chrisgebert commented 7 months ago

delete + insert seems to be the choice to overwrite existing records that have been updated since a previous snapshot file. I'll need to confirm the columns that should be included/excluded.