RowCount / SumCheck Tests based on Historical Averages / St.Devs.

[DovileKr]

Is your feature request related to a problem? Please describe. Current Table Tests are somewhat limited in their context and ability to provide value. For example:

• Table Row Count to be Between - only adds value if checked against historical averages. For example, on average we have 100k daily transactions. Expected fluctuation is 30%. Today was loaded 20k transactions - Failed. These thresholds change due to seasonality or business growth or decline.
• Table Column Count to Equal - only adds value if comparison is against "previous load" value. For example, yesterday had 10 columns, today 9. Failed.

Describe the solution you'd like I do not know how, but ideally table metrics would be tracking daily changes on RowCount or CheckSum (for pre-selected columns), deriving averages. Then allowing to choose threshold (like -/+ 20%, or ideally St.Dev. with confidence level). And then setting this as dynamic Min/Max values for RowCounts or Column Sums.

This would be a very powerful data validation test, where even before loading, data pipeline could double check with OMD - "is my value" within expected ranges.

Describe alternatives you've considered

• Use partitioned profile, for example on "sales date" or "load date"
• Calculate daily / or periodic rowcount / sumcheck values outside OMD and update testCases with the new Table Row Count to be Between values or Perform such validations as part of data pipeline and just submit the results to OMD, without parameters.

Additional context Add any other context or screenshots about the feature request here.

[TeddyCr]

Hey @DovileKr, thank you for suggesting this feature. This is actually already in the roadmap but for Collate SaaS/OnPrem (and not the Open Source version) as part of our anomaly detection feature. I'll mark this issue as won't fix for now.