dbt-snowflake needs a microbatch implementation that:
efficiently inserts new batches of data knowing that compiled_code will be filtered down by event_time
does not require a unique_key configuration on the model.
Solution
Initially, this was implemented with the delete+insert strategy already defined for dbt-snowflake. However, this creates a requirement for snowflake microbatch models to specify a unique_key, which is actually not strictly necessary!
Create a custom strategy implementation that simply:
Deletes the previous partition of data (using __dbt_internal_microbatch_event_time_start and __dbt_internal_microbatch_event_time_end
Inserts the new data from the temp table, since it is filtered-down already via compiled_code
Testing:
override the insert_two_rows_sql fixture to use snowflake-specific syntax
override the microbatch_model_sql fixture to demonstrate the lack of requirement of a unique_key config
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide.
resolves #1182 docs dbt-labs/docs.getdbt.com/# N/A
Problem
dbt-snowflake needs a microbatch implementation that:
compiled_code
will be filtered down by event_timeunique_key
configuration on the model.Solution
unique_key
, which is actually not strictly necessary!compiled_code
Testing:
insert_two_rows_sql
fixture to use snowflake-specific syntaxmicrobatch_model_sql
fixture to demonstrate the lack of requirement of aunique_key
configChecklist