Benchmarking module

[fredmontet]

Loading data

• [ ] some basic datasets
• [ ] custom dataset
• [ ] TimeEval format (https://timeeval.readthedocs.io/en/latest/)

Model training

• [ ] encapsulation of models to be tested in a dedicated object (e.g. ModelToBenchmark )
• [ ] possibility of running the benchmark on a series of datasets
• [ ] optimization with Optuna using e.g. a boolean

Comparison for Predictors

• [ ] Test of multiple predictors and select the best one (check prediction quality)
• [ ] Performance plot for each model and dataset
• [ ] Comparison of characteristics of selected models (confidence interval, point prediction or not, etc.)
• [ ] Test of multiple prediction windows and select the best one
• [ ] Test at different time along the tested window

Comparison for Detectors

• [ ] Test of multiple statistical detection thresholds and select the best one

[fredmontet]

1. Chargement des donnée
2. POC de module de benchmark (MVP test de 2 modèle, 2 datasets) MS1 - Validation archi
3. Ajout de features (optimisation, longueur de prediction, features de modèle) MSx - feature x
4. Reporting et format des résultats (.md, .png, .pdf, exécution hors de notebooks de préférence)