Closed lilipopololo closed 3 years ago
Hello, I remember that the loss multiplied by 15 was commented which was not used in the training. It was originally used during debugging.
The corresponding code in JDACS is:
self.unsup_loss = 12 * self.reconstr_loss + 6 * self.ssim_loss + 0.18 * self.smooth_loss
.
The corresponding part in JDACS-MS is:
self.unsup_loss = 12 * self.reconstr_loss + 6 * self.ssim_loss + 0.05 * self.smooth_loss
.
In general, the training loss should decline under the default setting.
You may find out that the weight of the self.smooth_loss
is modified according to different frameworks and backbones.
The reason is that the smooth_loss
term may hinder the convergence of self-supervised training if an inappropriate weight is used.
In intuition, in the early stage of training, if the smooth_loss
is too large, which provides a strong regularization to self-supervised training, the training loss may fall into a trivial solution rather can converge to a stable status.
As a result, the trade-off between each term in self-supervision requires cautious alteration.
Could you provide the curve of each component in the training loss, especially the smooth_loss
?
I have tried to train this net when I set batchsize==8 and trained in 20 epoch,but I can't understand why the training loss don't decline and surrounding 10%. I can see the loss is in losses/unsup_loss and combine with a Vgg pretrain block ,the loss is
12 * self.reconstr_loss+6 * self.ssim_loss + 0.05 * self.smooth_loss (0.8*x+0.2*y+0.05*z)*15
I can't understand why the loss must multiply by 15.
I hope somebody can explain this,3U