search

calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
989 stars 120 forks source link

add basic column tests for arrays and update readme #273

Open ilkernator opened 11 months ago

ilkernator commented 11 months ago

Issue this PR Addresses/Closes

272

Summary of Changes

Added two new tests to the "column_values_basic" directory and updated the readme-file:

Why Do We Need These Changes

For tables that contain columns of type array or array(struct), the values can be sometimes empty without being NULL (e.g. [ ], or [" "]. In this case, the test to expect_column_values_to_not_be_null.sql could be misleading and mask the issue of having empty values. So we would need to make use of Hive functions to assert the array is indeed empty.

Reviewers

@clausherther

clausherther commented 10 months ago

Hi, thanks for this PR. Before I can take a look at this, we'll need to add integration tests for both macros, which requires adding the appropriately typed columns to the test datasets. Also, I think I'd prefer implementing this more generically as a single test, ~ expect_column_to_not_be_empty. Would that be possible across all platforms we support? Also, we'd need the positive equivalent, expect_array_column_to_be_empty.