Open kdallmeyer-sr opened 3 years ago
An idea to solve this:
Collect these tests in it's own sequence or list that are returned to the topmost data description. Then if present, as it reads each data element it creates an internal map of datatrace to summary statistic for that value. For simplicity the summary map would be one giant flat one. Then apply the dataset tests to the summary map at the end.
This would mean that the CSV validator would have to be modified so that row number is no longer a part of the datatrace. Instead it should be a tag like Json and Xml validators.
A common data validation task is to compare summary statistics. For example, whether the expectation is that the average price of a product is near certain defined mean or in a defined std dev range. Likewise someone might want to compare averages across two attributes.
Typically this would be done in SQL after the data is at rest. It would be great if Baleen supported both out-of-box statistical aggregations like
sum
,mean
,variance
,stddev
,min
,max
, andcount
. Likewise, Baleen should support custom aggregations that can be done across large datasets.Additionally I think it would be interesting to provide a utility function that calculates summary statistics across the entire dataset. Something like