historical_forecast requires shift

[JulianNyarko]

When trying to validate an NBEATSModel model trained via historical_forecast with small input_chunk_length and output_chunk_length, it seems the prediction is always one step off. For instance, on simulated data, I run:

model_NBEATS = NBEATSModel(input_chunk_length=1, output_chunk_length=1, n_epochs=50, torch_device_str="cuda:1")

hf_cov = model_NBEATS.historical_forecasts(ts_list_all[0],
                                              past_covariates=cov_list_all[0],
                                              start=30,
                                              forecast_horizon=output_size,
                                              stride=1,
                                              retrain=True)

When evaluating performance like this

from darts.metrics import rmse
rmse(ts_list_all[0], hf_cov)

I get 1.64

But when evaluating performance with this

rmse(ts_list_all[0], hf_cov.shift(-1))

I get 1.04. This is consistent across different target series and also other models that use past covariates (like TCNModel). I have confirmed this through the plots, which make it pretty obvious predictions are exactly one step off. When increasing the target series (from 50 to 500 periods) and setting input_chunk_length and output_chunk_length sufficiently high (e.g. 30 and 15), the predicted series is the most accurate and does not require a shift.

To make sure it is not my data, I went to this blog post. When training an NBEATSModel and setting input and output chunk length sufficiently small, the issue replicates on that data as well.

I understand setting input_chunk_length and output_chunk_length to small values is not super common, but the predictions are actually very good once they are lagged by one period. Any suggestions on what might be going on?

[JulianNyarko]

I realize this might just mean that the model doesn't update its predictions much, given an input. If that's all there is to it, then it seems to just be a small data problem. Will close for now!

[swarmt]

Hi Julian,

I am seeing the same thing when training with output_chunk_length=1. My gut feeling is that the model is basically producing a naive forceast.