[paper] && [code] Authors:
Qizhe Xie∗ 1, Minh-Thang Luong1, Eduard Hovy2, Quoc V. Le1
1Google Research, Brain Team, 2Carnegie Mellon University
Highlight
This paper presents Noisy Student Training which extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. This method achieves state-of-the-art results on ImageNet images. And experiments on ImageNet-A, ImageNet-C, and ImageNet-P show the model's robustness.
Basically, Noisy Student Training has three main steps: (1) train a teacher model on labeled images (ImageNet), (2) use the teacher to generate pseudo labels on unlabeled image, and (3) train a student model on the combination of labeled images and pseudo labeled images. They iterate this algorithm a few times by treating the student as a teacher to relabel the unlabeled data and training a new student.
Main Contributions
ImageNet SOTA
Used extra dataset.
But, order of magnitude less dataset (3.5B -> 300M) compared to previous SOTA
But, unlabeled dataset, instead of weakly labeled. (Use teacher model to generate pseudo labels.)
[paper] && [code]
Authors:
Qizhe Xie∗ 1, Minh-Thang Luong1, Eduard Hovy2, Quoc V. Le1 1Google Research, Brain Team, 2Carnegie Mellon University
Highlight
This paper presents Noisy Student Training which extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. This method achieves state-of-the-art results on ImageNet images. And experiments on ImageNet-A, ImageNet-C, and ImageNet-P show the model's robustness.
Basically, Noisy Student Training has three main steps: (1) train a teacher model on labeled images (ImageNet), (2) use the teacher to generate pseudo labels on unlabeled image, and (3) train a student model on the combination of labeled images and pseudo labeled images. They iterate this algorithm a few times by treating the student as a teacher to relabel the unlabeled data and training a new student.
Main Contributions