[QUESTION] Validation metric per fold

[guimalo]

Is there a way to automatically get the value of the evaluation metric per fold during backtesting? Something like what sktime's evaluate does?

[JavierEscobarOrtiz]

Hello @guimalo,

In backtesting, the metric is calculated after all fold predictions have been generated. The code is something like this:

mean_absolute_error(
    y_true = y.loc[backtest_predictions.index],
    y_pred = backtest_predictions['pred']
)

In backtesting, folds are created to mimic what would happen if the model were run in production (availability of historical data and predicted steps). For each fold, predictions are made and stored, but no metric is calculated. Once the backtesting process is complete, the predictions of each fold are concatenated (since the folds are temporally ordered) and the metric is calculated for the entire period.

If the backtesting period is the year 2024, the result would be: The metric of the model for the year 2024 is xxx.

If you want to calculate a metric per fold, I suggest you store the backtesting predictions and apply the metric functino you want by iterating over the index.

Hope it helps!