search

cvgmi / manifold-net-dmri

ManifoldNet Paper Implementation for SPD(n)
9 stars 1 forks source link

gradient exploding problem #2

Open zhenglong178 opened 3 years ago

zhenglong178 commented 3 years ago

when i try to use ManifoldNet to train my own dti data, i meet a problem with gradient exploding. I would like to ask if you have ever encountered gradient explosion problems in your training？Can you give some advice on this problem？Looking forward to your reply. Thank you. Questions as follows： name: mnet_model.conv1.weight_matrix -->grad_requirs: True -->grad_value: tensor([[ nan, nan, nan, ..., nan, nan, nan], [ 2.7896e-14, -1.7411e-12, 1.7570e-13, ..., 1.0343e-14, 2.6899e-14, -5.6835e-12]], device='cuda:0')

Invincible-sin commented 2 years ago

I have also encountered situations where nan appears, of course, provided that you can get an example like the author gave. You can try to add a line of code after each "torch.matual", such as "a=(a+a.permute(0,2,1)/2" to ensure that the matrix obtained each time is symmetric and positive definite, so It can solve part of the situation where nan appears, but it is not guaranteed to solve your problem.