[QUESTION] training data does not match training data prediction

[benjamin-strath]

Hi there,

I have been using the darts TFT model for some time now and have come across an issue. To check whether my models have been trained correctly I make predictions on the training data to see how they compare to the true training data. For training data I would expect an almost exact match between predictions and true data however I have noticed this is not always the case, even when training the model for a large number of epochs. I have also noticed that this problem reduces when more and more timeseries are used to train the model. I am unsure why this is happening and wondered if anyone else has had a similar experience. I also wondered if darts has some method to prevent overfitting or something like that which is causing the discrepancy?

For some context I am using the TimeSeries.from_group_dataframe to generate timeseries from my data. I am using multiple timeseries to train my model (global forecasting). And I am using a mixture of future and static covariates.

Any help would be greatly appreciated, thanks!

[madtoinou]

Hi @benjamin-strath,

So to assess if your model was fitted appropriately, you try to forecasts the training set and assess if the predictions match the data perfectly? This should be achievable if a single series is used to fit the model (especially if you considerably increase the number of parameters), however, by training it on several series, the model should generalize better and not over-fit one series by definition (which is actually desirable).

I have also noticed that this problem reduces when more and more timeseries are used to train the model. I am unsure why this is happening and wondered if anyone else has had a similar experience.

I suspect your time-series to share a lot of pattern/seasonality (making them very suitable to global training) if you observe that the predictions on the training data get better and better as you increase the number of series used during training. This is actually great; your model generalize so well that it's also excellent at forecasting the training set.

Finally, Darts does not implement any logic to avoid overfitting in torch-based models by default, the users are responsible for it.