NBEATS missing Early Stopping Callback.

[fornasari12]

I'm currently working with your NBEATS implementations, and I couldn't find any option to set an Early Stopping Callback.

I'm missing sth, or that is not available for training?

Is there any option for implement it?

Thks!

[pennfranc]

Hey @fornasari12, If you simply want the model that achieved the best performance on the validation set, you can do this: Instantiate your NBEATSModel instance with the save_checkpoints parameter set to true. Then after having trained your model, you can call the NBEATSModel.load_from_checkpoint() method with best=True to get the specific model parameters that produced the lowest error in the validation set during training.

However, this does not give you the time-saving aspect of early stopping. In general, we currently do not have the option to provide early stopping callbacks. If you're up for it, we would very much appreciate your contribution! The file you would have to change is this one: https://github.com/unit8co/darts/blob/master/darts/models/forecasting/torch_forecasting_model.py

Let me know if you have any other questions.

[fornasari12]

Hello @pennfranc, thanks for your response!!

I'll try with your suggestions first, then I'll try to add de early stoping callback.

One last question, currently I'm working on forecasting for around 400k time series with hourly freq, you think this implementation of NBEATS would be able to handle this amount of series?

Thanks in advance!

[hrzn]

Hi @fornasari12, it is hard to know in advance how the model will do on a given dataset, but some experiments have shown NBEATS to work very well on several thousands of time series. Our implementation is not special and should scale well. On large datasets it will help if you can use a GPU for training, and apply some performance tuning tricks. I have written a guide on performance tuning for deep learning models with Darts here: https://github.com/unit8co/darts/blob/master/guides/performance.md It relies on some features that will only be released in the upcoming v0.14.0, though.

[hrzn]

I'll close this issue for the time being and we'll use https://github.com/unit8co/darts/issues/577 to track the early-stopping topic instead.

[maheshs11]

Hello @pennfranc, thanks for your response!!

I'll try with your suggestions first, then I'll try to add de early stoping callback.

One last question, currently I'm working on forecasting for around 400k time series with hourly freq, you think this implementation of NBEATS would be able to handle this amount of series?

Thanks in advance!

best model not found error @hrzn

[fornasari12]

Hi @fornasari12, it is hard to know in advance how the model will do on a given dataset, but some experiments have shown NBEATS to work very well on several thousands of time series. Our implementation is not special and should scale well. On large datasets it will help if you can use a GPU for training, and apply some performance tuning tricks. I have written a guide on performance tuning for deep learning models with Darts here: https://github.com/unit8co/darts/blob/master/guides/performance.md It relies on some features that will only be released in the upcoming v0.14.0, though.

Thks @hrzn , I'll give it a try!