Open xmy0916 opened 2 years ago
Yes, you may be right, we can try to change 1e-8 to a large one, did you try it?
@yuanli2333 I have tested 1e-4 but also can't fix the problem.
@xmy0916 In my implementation, 1e-6 can fix this problem, with bs=2048 and lr=1e-3.
Thanks!
https://github.com/yitu-opensource/T2T-ViT/blob/main/models/token_performer.py#L18 My code has turned on fp16, so the 1e-8 on this line to prevent division by 0 is not enough for my code... the loss of the network calculation appears nan due to this code : https://github.com/yitu-opensource/T2T-ViT/blob/main/models/token_performer.py#L50