calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
989 stars 120 forks source link

[Feature Request] Fast check that table is not empty #269

Open bochkarevnv opened 1 year ago

bochkarevnv commented 1 year ago

Is your feature request related to a problem? Please describe. I need to check that table have any rows, but I don't see good expectation for that.

Describe the solution you'd like I want fast probe if table has one row. If you want to check table not empty - you don't need to count all rows in table.

Describe alternatives you've considered You can use expect_table_row_count_to_be_between with min_value 0, but it counts all rows in table, it can be very slow.

Additional context In my case I write test with limiting number of rows selected for count. I think you can implement something like this.

{%- test expect_table_non_empty(model, row_condition=None) -%}
{{ default__test_expect_table_non_empty(model, row_condition) }}
{% endtest %}

{%- macro default__test_expect_table_non_empty(model, row_condition) -%}
with
    limited as (
        select 1
        from {{ model }}
        {%- if row_condition %} where {{ row_condition }} {% endif %}
        limit 1
    ),
    grouped_expression as (select count(*) as expression from limited),
    validation_errors as (select * from grouped_expression where not (expression > 0))

select *
from validation_errors

{%- endmacro -%}