search

business-science / modeltime.gluonts

GluonTS Deep Learning with Modeltime
https://business-science.github.io/modeltime.gluonts/
Other
39 stars 9 forks source link

Error NBEATS : Epoch[0] gave NaN loss and it will be ignored #26

Open mdancho84 opened 3 years ago

mdancho84 commented 3 years ago

I'm trying to run a version with the "Alternative Padding Strategy" suggested in 18.1. Everything has worked well until Model 6 NBEATS ensemble. I've tried adjusting the nbeats() arguments, but the workflow still crashes. Have you seen the error below before? I found a similar warning online for MQCNNEstimator, not for NBEATSEstimator or NBEATSEnsembleEstimator. 

> wflw_fit_nbeats_6_2 % 
+    add_model(model_spec_6_nbeats) %>% 
+    add_recipe(recipe_spec_gluon_2) %>% 
+    fit(training(splits_2))
  0%|          | 0/35 [00:00, ?it/s]WARNING:gluonts.trainer:Batch [2] of Epoch[0] gave NaN loss and it will be ignored
vidarsumo commented 3 years ago

I've had this issue with DeepAR and then it was due to exploding gradients and I had to adjust the clip_gradient. I don't know the exact architecture of N-BEATS, but could this also be the case here?