search

Mukosame / Zooming-Slow-Mo-CVPR-2020

Fast and Accurate One-Stage Space-Time Video Super-Resolution (accepted in CVPR 2020)
GNU General Public License v3.0
917 stars 164 forks source link

Warning：Offset mean is XXX，larger than 100 #30

Closed porcofly closed 4 years ago

porcofly commented 4 years ago

2020-08-03 15-37-21屏幕截图 2020-08-02 17-37-36屏幕截图

Thank you for open source your code, when train the model on Vimeo 90k，this warning occurred. It first appeared after the first decay of the learning rate, and then the offset mean will become larger and larger. Have you ever encountered this situation? Is this normal in your training?

Mukosame commented 4 years ago

Hi @porcofly , basically this warning will not kill your training process, but you have to be careful of it --- since the Vimeo90k is only ~480 in size, having a 200 offset probably means your model doesn't converge. I also encountered such situations before. In fact, such issues also happen to other works that use DCN. You can try some tricks like warm-up, or initialize with our pretrained weights to provide a safer start point for you model to prevent it from happening again. Hope these can solve your questions!

porcofly commented 4 years ago

Hi @porcofly , basically this warning will not kill your training process, but you have to be careful of it --- since the Vimeo90k is only ~480 in size, having a 200 offset probably means your model doesn't converge. I also encountered such situations before. In fact, such issues also happen to other works that use DCN. You can try some tricks like warm-up, or initialize with our pretrained weights to provide a safer start point for you model to prevent it from happening again. Hope these can solve your questions!

Thanks for your reply, I changed the learning rate to 1 e-4 and train from scratch.Then this problem seem be solved