calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
1.06k stars 130 forks source link

ignore_row_if not adding to test query in expect_compound_columns_to_be_unique #170

Closed prachishah48 closed 2 years ago

prachishah48 commented 2 years ago

https://github.com/calogica/dbt-expectations/blob/6bd8b64d0978e59fc36dd36972beb37ce18d780a/macros/schema_tests/multi-column/expect_compound_columns_to_be_unique.sql#L21

We noticed that when we set the ignore_row_if argument to any of the options (any_value_is_missing or all_value_is_missing or never). It does not add a where statement to the validation_errors CTE.

We are not seeing the expected behavior for this macro/test.

Screen Shot 2022-04-19 at 2 38 47 PM Screen Shot 2022-04-19 at 2 38 39 PM
clausherther commented 2 years ago

Ah good catch @prachishah48 ! I think it's b/c row_condition_ext is set but never used 😭 . Should be able to fix that by tomorrow and cut a new patch release soon after. Is that bug blocking you from something?

prachishah48 commented 2 years ago

Hi @clausherther, Thank you for getting back to me :D and Nope, it is not blocking me from something but do let me know when it gets produced so I can see how the tests look

clausherther commented 2 years ago

I have PR out (#171 ), and the generated SQL in the test suite now looks like this for all_values_are_missing case

with validation_errors as (

    select
        date_col,col_string_b
    from data_test
    where
        1=1
        and 
        (
            date_col is not null and 
            col_string_b is not null

        )

    group by
        date_col,col_string_b
    having count(*) > 1

)
select * from validation_errors

and like this for the any_value_is_missing case

with validation_errors as (

    select
        date_col,col_string_b
    from data_test
    where
        1=1
        and 
        (
            date_col is not null or 
            col_string_b is not null

        )

    group by
        date_col,col_string_b
    having count(*) > 1

)
select * from validation_errors

And like this is an additional row_condition: 2=2 is specified:

with validation_errors as (

    select
        date_col,col_string_b
    from postgres.dbt_expectations_integration_tests.data_test
    where
        1=1
        and 
        2=2 
        and
        (
            date_col is not null and 
            col_string_b is not null

        )

    group by
        date_col,col_string_b
    having count(*) > 1

)
select * from validation_errors
prachishah48 commented 2 years ago

Oh great! Thank you!!