search

jdb78 / pytorch-forecasting

Time series forecasting with PyTorch
https://pytorch-forecasting.readthedocs.io/
MIT License
3.85k stars 609 forks source link

DeepAR may have information leaking #640

Open SonghuaHu-UMD opened 3 years ago

SonghuaHu-UMD commented 3 years ago

Description

Thanks for constructing such an excellent tool. I am using it to compare several models including TFT, DeepAR, N-BEATS, LSTM, GRU. Using the same dataset and fixed encoder length, I increase the prediction horizon (i.e. the decoder length) from 1 day to 21 days. I assume the loss should also increase with the increase of prediction horizon, however, the DeepAR did not work in that way. I listed the loss as follows:

  Loss Loss Loss Loss Loss
Name All-TFT Blank-DeepAR Blank-Nbeats Blank-RNN-GRU Blank-RNN-LSTM
Step          
1 0.085470483 7.008180618 0.087804154 0.097979955 0.09866339
7 0.100440644 6.953411102 0.110086426 0.108908094 0.112639382
14 0.116680592 6.927832127 0.126026124 0.118239745 0.121749699
21 0.112808555 6.991565228 0.134752601 0.127701327 0.126969561

All other models work perfectly but not DeepAR. I also found a previous issue related to this bug: https://github.com/jdb78/pytorch-forecasting/issues/182

From my experiment, I think it has not been solved. Any suggestions on this issue?

Really appreciated!

jdb78 commented 2 years ago

Looks indeed strange but might be simply a problem with statistics. How large is your dataset, e.g. for the TFT, I also see a decrease here from 14 to 21 steps.

ozanozyegen commented 2 years ago

Can you show the error rates of the models in the table? The training of deepar is autoregressive, and the true target values are provided (see figure 2 of the deepar paper). It might be normal for the training loss to stay stable, but when you predict with monte-carlo sampling, the validation loss should increase as you increase the prediction horizon.