search

liyaguang / DCRNN

Implementation of Diffusion Convolutional Recurrent Neural Network in Tensorflow
MIT License
1.22k stars 400 forks source link

train loss first fall and then rise #60

Closed meirena123 closed 3 years ago

meirena123 commented 3 years ago

When training the model in the 23rd epoch， train loss always starts to rise。 like this： 2021-04-13 16:24:03,766 - INFO - Epoch [22/100] (8625) train_mae: 1.9640, val_mae: 2.7854 lr:0.001000 362.1s 2021-04-13 16:24:03,798 - INFO - Val loss decrease from 2.7869 to 2.7854, saving to data/model\dcrnn_DR_2_h_12_64-64_lr_0.01_bs_64_0413104940/models-2.7854-8625 2021-04-13 16:30:05,780 - INFO - Epoch [23/100] (9000) train_mae: 1.9654, val_mae: 2.7894 lr:0.001000 362.0s 2021-04-13 16:36:08,940 - INFO - Epoch [24/100] (9375) train_mae: 1.9693, val_mae: 2.7856 lr:0.001000 363.2s 2021-04-13 16:42:11,874 - INFO - Epoch [25/100] (9750) train_mae: 1.9701, val_mae: 2.7837 lr:0.001000 362.9s 2021-04-13 16:42:11,908 - INFO - Val loss decrease from 2.7854 to 2.7837, saving to data/model\dcrnn_DR_2_h_12_64-64_lr_0.01_bs_64_0413104940/models-2.7837-9750