godatadriven / pytest-dbt-core

Pytest plugin for dbt core
https://pytest-dbt-core.readthedocs.io/en/latest/
Apache License 2.0
54 stars 2 forks source link

[Feature] Support dbt 'is_incremental' in pytest-dbt-core #24

Open danaeder opened 1 year ago

danaeder commented 1 year ago

Is this your first time opening an issue?

Describe the Feature

dbt adds the 'this' somehow in the is_incremental macro: https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/models/incremental/is_incremental.sql is_incremental.sql

{% macro is_incremental() %} {#-- do not run introspective queries in parsing #} {% if not execute %} {{ return(False) }}

Describe alternatives you've considered

I would like to test the following macro that uses 'is_incremental' and 'this' argument

{% macro get_table_max_id(where_clause = '', id_column = '"id"') %}
    {% if is_incremental() %}
        {% set max_id_query %}
        select coalesce(max({{ id_column }}), 0) from {{ this }}
        {{ where_clause }}
        {% endset %}
        {% set max_id_res = run_query(max_id_query) %}
    {%endif%}
    {{ return(max_id) }}
{% endmacro %}

following error: def raise_compiler_error(msg, node=None) -> NoReturn:

  raise CompilationException(msg, node)

E dbt.exceptions.CompilationException: Compilation Error in macro is_incremental (macros/materializations/models/incremental/is_incremental.sql) E 'this' is undefined E
E > in macro get_table_max_id (macros/unioner.sql) E > called by macro test_table_max_id (macros/test_macro.sql) E > called by macro is_incremental (macros/materializations/models/incremental/is_incremental.sql)

Who will this benefit?

All dbt users since is_incremental is very popular

Are you interested in contributing this feature?

No response

Anything else?

No response

JCZuurmond commented 1 year ago

Hi @danaeder, thanks for opening the issue. I see the this is missing; dbt adds this somehow; this package does not (yet).

Could you try the following for me?

{% macro get_table_max_id(where_clause = '', id_column = '"id"') %}
        {% set max_id_query %}
        select coalesce(max({{ id_column }}), 0) from {{ this }}
        {{ where_clause }}
        {% endset %}
        {% set max_id_res = run_query(max_id_query) %}
    {{ return(max_id) }}
{% endmacro %}

What happens with the this after the from (third line)? Is that compiled correctly now?

And, something that might unblock you quicker, is it possible to use the is_incremental outside the macro. I expect that you have something like:

SELECT * 
FROM my_table 
{{ get_table_max_id() }}

What happens if you move the is_incremental out of the macro:

SELECT * 
FROM my_table 
{% if is_incremental() %}
{{ get_table_max_id() }}
{%endif%}