Backtesting a NeuralForecast model with re-training

[MLfreakPy]

Hi together,

I want to backtest a NBeats model with re-training. Hence, expanding window strategy, simulating the inflow of new data every week, with weekly re-training and subsequent forecasting.

I am aware of the cross_validation method, however, my impression is that it is not re-training the model per step, just making predictions based on a one-time-only fitted model.

Aim: having a train set, val set, as well as a combined set: train over train set, start making predictions and evaluating the model over the val set. Re-train and predict every 5 days, expanding the window continuously until end of val set.

Having just recently moved over to using Nixtla, I have screened extensively through the tutorials and the web, but could not find it.

Do you know of a possibility to do so?

Your help is highly appreciated!

Best Steffen

---- Current code base

## Define Models
model = [NBEATS(h=5, 
                input_size=12,  
                max_steps=max_training_steps, 
                scaler_type='standard')                             
                ]

## Instantiate models
nf = NeuralForecast(
    models=model,
    freq='B')

## Run cross-validation
stride = 5                 

cv_df = nf.cross_validation(
    df = nxt_ts_train_val_t,
    step_size=stride,
    n_windows= 3,  
    use_init_models= False, 
    )   

[ams015]

I am doing the same thing. I noticed that someone asked a question in March about cross validation with retraining(but not from scratch), and the answer was they don't seem to have that as a function. If that hasn't changed then you would need to make a function yourself. https://github.com/Nixtla/neuralforecast/issues/460

[MLfreakPy]

Thank you very much! I also raised the question in the slack community. I just got the response that, it is indeed not an inbuilt functionality yet, but they are currently working on it :-) So I will be building my own for-loop as well. Thanks a lot ams015 and happy coding!

[ams015]

You're welcome! I have been using these 2 for generating the indices for the train/test in each window https://alkaline-ml.com/pmdarima/modules/generated/pmdarima.model_selection.RollingForecastCV.html https://alkaline-ml.com/pmdarima/modules/generated/pmdarima.model_selection.SlidingWindowForecastCV.html