dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
297 stars 119 forks source link

Allow creation of snowpipes without full refresh #158

Closed mirkoprescha closed 1 year ago

mirkoprescha commented 2 years ago

In case you have a large warehouse you don't want to import all existing files into the snowpipe table. Only after using --vars "ext_full_refresh: true" all the files linked to the snowflake stage will be imported. Each dbt run-operation stage_external_sources will create snowpipe(s) with create or replace. Currently, we can not use snowpipes with DBT on large s3 folders because dbt creates only a snowpipe when the table does not exist or with the applying --vars "ext_full_refresh: true". However, in both cases happens always a full copy of existing files.

Description & motivation

Checklist

github-actions[bot] commented 1 year ago

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

github-actions[bot] commented 1 year ago

Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers.