search

CuriousAI / mean-teacher

A state-of-the-art semi-supervised method for image recognition
https://arxiv.org/abs/1703.01780
Other
1.58k stars 331 forks source link

About two different loss while in training progress #62

Open DISAPPEARED13 opened 2 years ago

DISAPPEARED13 commented 2 years ago

Hi there, I download this code and adapted them for my semi-supervised segmentation (Pytorch version). And thanks for this genius code you provided!

But I have a question is that I know that mean-teacher model contains two loss, one is for unsupervised loss for paired labeled data and ground-truth, and the other is for contrast loss.

And here is the contrast loss I calculated:

  1. get the consistency weight by 10 * sigmoid_rampup(epoch, 5) at each epoch
  2. compute the logits from student and teacher's output

And then I got contrast loss up to thousands and it seems not right. Is it normal or some bug in my code? Could you give me some advice if you have some idea? Thanks!

DISAPPEARED13 commented 2 years ago

Hi, there. I am here to update my problem. For debugging and I found that the problem is from softmax_mse_loss after change it to nn.MSEloss() the problem of loss up to thousands is gone.

But it's still confused me that,

will contrast loss is the regularization for student model and teacher model, so it won't decrease too low or increase too much, just oscillate around e-4 or something else?

Thanks.