search

luhaofang / tripletloss

tripletloss in caffe
MIT License
322 stars 152 forks source link

fc9_1 weights is increase with the train iterations from 0.0x to 40.x #53

Open zhouyongxiu opened 6 years ago

zhouyongxiu commented 6 years ago

@luhaofang Hi, when I use the code to train the fc9_1, I find that the fc9_1 weights is increase with the train iterations, after train 40000, it increase from 0.0x to 40.x, do you know what to lead this problem, and how to deal it.

gombru commented 6 years ago

Hi, I had a similar problem. Weights increased while the net was learning until at some point I got NaNs. I solved it using a smaller margin.

Though still I haven't analyzed it deeply, I think that the reason is that when the margin is too big compared to the magnitudes of the vectors you are comparing (in my case they were L2 normalized) the net cannot achieve to produce those big distances and weights keep going up.