search

YU1ut / MixMatch-pytorch

Code for "MixMatch - A Holistic Approach to Semi-Supervised Learning"
MIT License
633 stars 129 forks source link

about interleave labeled and unlabed samples #5

Closed blankWorld closed 5 years ago

blankWorld commented 5 years ago

Thanks for your implementation ! Why interleave labeled and unlabed samples between batches can get correct batchnorm calculation.

Wondersui commented 5 years ago

I have the same question

YU1ut commented 5 years ago

Batchnorm estimates the mean and std of the whole dataset by each batch. Since we feed labeled batch and unlabeled batch containing the same number of samples alternately, this estimation may be biased when the number of labeled samples is very small.

blankWorld commented 5 years ago

You are right, thanks for your answers, and I have another question. In one iteration of training, batchnorm layers are updated three times and each update is completed with 64 images. Why not update batchnorm layers one time using total 64*3 images in one iteration of training？

YU1ut commented 5 years ago

Yes, you can do that. But the learning rate should be adjusted according to the batch size. I just try to follow the official implementation in this repository.

blankWorld commented 5 years ago

Thanks again for your kindness. I have already reproducted the official results.

JongMokKim commented 5 years ago

Hi blankWorld, did u make any change code to reproduce official result? or did u get with this code? Thank u in advance!