nlpyang / PreSumm

code for EMNLP 2019 paper Text Summarization with Pretrained Encoders
MIT License
1.29k stars 465 forks source link

Continuous Training Performance Drop #154

Open YiSyuanChen opened 4 years ago

YiSyuanChen commented 4 years ago

Hi all, Firstly, I would like to thank the authors for giving such excellent works. I have some questions about the training that starts from checkpoint. Below is my validation xent curve, and different color indicates the boundary for continuous training.

截圖 2020-04-15 下午9 28 33

The setting is identical in the training process, but the performance drops suddenly at the beginning of continuous training. I also check the training xent curve and the learning rate curve to make sure the optimizer is also loaded correctly:

截圖 2020-04-15 下午9 28 24 截圖 2020-04-15 下午9 28 40

As you may see, the training seems to be normal. Thus, I would like to search for the help. Does anyone suffer this issue before? Any comment would be very appreciated.