dbt-labs / dbt-adapters

Apache License 2.0
25 stars 35 forks source link

[Feature] Custom Incremental Strategies Need Access to Config #228

Closed tnightengale closed 4 months ago

tnightengale commented 4 months ago

Is this your first time submitting a feature request?

Describe the feature

Custom incremental strategies are the suggested approach to extending the incremental materialization.

However, they are stunted by an inability to utilize values from the model config. This is because the default incremental materialization only passes a limited dictionary,strategy_arg_dict, to the incremental strategy:

    {#-- Get the incremental_strategy, the macro to use for the strategy, and build the sql --#}
    {% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
    {% set incremental_predicates = config.get('predicates', none) or config.get('incremental_predicates', none) %}
    {% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}
    {% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': temp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'incremental_predicates': incremental_predicates }) %}
    {% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}

A seemingly better choice would be to also pass the model config in it's own key, so as to extend the current approach, without requiring extensive refactoring.

This would allow implementing a custom strategy that can rely on the model config, eg:

{% macro default__get_incremental_STRATEGY_sql(arg_dict) %}

  {% do return(get_CUSTOM_sql(
                    arg_dict["target_relation"],
                    arg_dict["temp_relation"],
                    arg_dict["unique_key"],
                    arg_dict["dest_columns"],
                    arg_dict["incremental_predicates"],
                    arg_dict["config"] -- Notice config can be unpacked here
                )
) %}

{% endmacro %}

{% macro get_CUSTOM_sql(target, source, unique_key, dest_columns, incremental_predicates, config) -%}
  {{ adapter.dispatch('get_CUSTOM_sql', 'dbt')(target, source, unique_key, dest_columns, incremental_predicates, config) }}
{%- endmacro %}

{% macro default__get_CUSTOM_sql(target, source, unique_key, dest_columns, incremental_predicates, config) -%}

    {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}

    {%- if config.get('KEY') -%}
    {#- Do the custom thing. -#}

...

Describe alternatives you've considered

Using a custom implementation of the incremental materialization. However, this would be unwieldy to keep up to date with the implementation defined in this repository.

Who will this benefit?

Anyone who wants more control when creating custom incremental strategies.

Are you interested in contributing this feature?

Yes.

Anything else?

No response

tnightengale commented 4 months ago

Nevermind, config is globally scoped 😬