search

flexanalytics / dbt_observability

Apache License 2.0
1 stars 0 forks source link

Feature: add automatic anomaly detection for column stats #5

Open MiConnell opened 6 months ago

MiConnell commented 6 months ago

Currently, column statistics are stored based on each run (row_count, sum, avg, etc). but the only way to tell if a large change has occurred is visually through a BI tool or manually with a query. Flags with a threshold would be very useful.

Something like the below variable

"dbt_observability.column_anonamly_threshold_percent": "5"

which would then flag any time that a column stat changed by more than 5%.

ataft commented 3 months ago

Perhaps this could be done similar to https://github.com/flexanalytics/dbt_observability/pull/28, but with a test on the column rather than the model