Benchmarking Minimum Viable Product

[fredmontet]

Goal

Develop the minimum viable product (MVP) of a benchmarking tool

Description

The benchmarking MVP should test two models with two datasets. The results should should be presented in a way that helps the choice of the best model (table, report in .md, plots, etc.) given appropriate metrics.

The development of this MVP might take 2-3 iterations to validate the benchmarking 'base features'. It will be the foundations for the other dimensions that the benchmarking module will be able to test e.g.

• model features (univariate, multivariate, confidence interval, required computation for training, etc.)
• performance given different prediction length
• performance given different prediction starting time

Tasks

• [x] Start the MVP of the module in a directory called benchmarking in https://github.com/fredmontet/ontime/tree/develop/src/ontime/module
• [x] Use the data module from issue #33
• [ ] Develop a Benchmark class with method such as add_model(my_model), add_dataset(), run(), etc.
• [ ] Benchmark 2 datasets with 2 models and present the result in a notebook