search

pkuxmq / Invertible-Image-Rescaling

[ECCV 2020, IJCV 2022] Invertible Image Rescaling
Apache License 2.0
630 stars 86 forks source link

The training is unstable #35

Open shengkelong opened 3 years ago

shengkelong commented 3 years ago

Thank you for your impressive work. But when I try to recurrent this network(I rewrite the code myself), sometimes the loss will suddenly increase by 10 times. The structure of the network is correct because I can load the pretrained network, so I think there may be some details I didn't notice. Could you tell me what methods you have taken in training to ensure stability?

pkuxmq commented 3 years ago

We restrict the range on exp() and apply gradient clipping. For reasons please refer to https://github.com/pkuxmq/Invertible-Image-Rescaling/issues/24.

Feynman1999 commented 1 year ago

Thank you for your impressive work. But when I try to recurrent this network(I rewrite the code myself), sometimes the loss will suddenly increase by 10 times. The structure of the network is correct because I can load the pretrained network, so I think there may be some details I didn't notice. Could you tell me what methods you have taken in training to ensure stability?

I have also encountered this issue recently, despite using gradient clipping strategy. How did you solve it?