search

databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
226 stars 119 forks source link

Validation on incremental strategies does not allow for custom strategies #783

Open Ruth-wj opened 2 months ago

Ruth-wj commented 2 months ago

Describe the bug

Custom strategies are supported in dbt-core and allow users to write macros that define incremental merge startegies ( https://docs.getdbt.com/docs/build/incremental-strategy#custom-strategies ). As long as the strategy macro is found in the namespace, it can be used.

The databricks adapter, however, overrides this functionality. A validation function is used which hardcodes only the builtin incremental strategies, constraining the user unnecessarily. The validation function is here: https://github.com/databricks/dbt-databricks/blob/6b29d329ae8a3ce6bc066d032ec3db590160046c/dbt/include/databricks/macros/materializations/incremental/validate.sql#L38

Personally I have overridden this validation function to include my custom merge strategies, but obviously this isn't ideal.

Steps To Reproduce

Implement custom merge strategy following dbt documentation

Expected behavior

Custom merge strategy should be usable provided they are in the namespace

benc-db commented 2 months ago

I'm fairly sure we imported/inherited this behavior from dbt-spark. At this point I have no idea why the limitation exists, so I will discuss with my team about lifting it.

mcowart123 commented 2 months ago

Same issue: 753

mcowart123 commented 1 month ago

Has there been any discussion had/movement on this?

benc-db commented 1 month ago

I'm in favor of removing the limitation, but haven't had time to comb the code to understand impact. There may be a couple of places where we assume a fixed set of strategies and make some assertions based on that.

ldnicolasmay commented 2 weeks ago

Same thing happens with BigQuery adapter:

19:44:52  Finished running 1 view model, 51 data tests, 2 incremental models in 0 hours 0 minutes and 8.93 seconds (8.93s).
19:44:52
19:44:52  Completed with 1 error and 0 warnings:
19:44:52
19:44:52    Compilation Error in model int_my_model (models/intermediate/int_my_model.sql)
  Invalid incremental strategy provided: insert_only
      Expected one of: 'merge', 'insert_overwrite'

  > in macro dbt_bigquery_validate_get_incremental_strategy (macros/materializations/incremental.sql)
  > called by macro materialization_incremental_bigquery (macros/materializations/incremental.sql)
  > called by model int_my_model (models/intermediate/int_my_model.sql)
19:44:52
19:44:52  Done. PASS=18 WARN=0 ERROR=1 SKIP=35 TOTAL=54