Tool for generating tabular/graphical drift outputs from passed data

[sgreenbury]

Aim: to convert the prototyping work from cases into a flexible drift report generating script.

Make a script that parses an input config and csv and outputs html/pdf tables and graphs reporting drift trends from the data.

• Input interface (csv, flags) and config file (e.g. toml)
• Code for the iterating through the data with some trend schema (e.g. year by year) to generate output measures
• Identify what the report outputs should exactly look like (e.g. tables of test statistic values, graphs of p-values)

Packages to consider using: unitreport

[sgreenbury]

Current implementation config takes as inputs:

• dataset that has a batch feature that can be used to split into reference and registered
• features and batches to calculate scores over
• scores to limit the scores computed

And outputs:

• Plot for each score passed on features and batches

Next steps:

• [x] Get coding passing pylint and mypy
• [x] Implement tag for score to indicate whether a statistic or p-value is output
• [x] Write table to file as csv
• [ ] Factor out drift calculations into separate methods
• [ ] Allow the same registry to be used for calculations so reference data need not be reloaded (see #6)
• [ ] Write graphs as a single pdf file

[sgreenbury]

Closing in favour of continued development in cases repo.