dbt-labs / dbt-audit-helper

Useful macros when performing data audits
https://hub.getdbt.com/dbt-labs/audit_helper/latest/
Apache License 2.0
329 stars 40 forks source link

bug fix: account for incorrect comparisons involving null values #67

Closed leoebfolsom closed 1 year ago

leoebfolsom commented 1 year ago

Description & motivation

Currently, the compare_all_columns macro does a poor job of distinguishing between missing data and null values.

We must check to see if the primary key exists in both of the two tables to determine whether the value in that table is null or missing.

With this update, the code checks that a primary key exists in both tables before declaring a perfect_match. This was already accounted for in other logical checks (e.g., null_in_a, null_in_b).

There are also errors in the conflicting_values logic leading to incorrectly counting conflicting_values as false when comparing a null to a not null value when the PKs are present in both tables.

Checklist

leoebfolsom commented 1 year ago

Issue/bug report: https://github.com/dbt-labs/dbt-audit-helper/issues/68