20NIPS # Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning

[XFeiF]

pdf
Code

Authors: Zhongzheng Ren∗, Raymond A. Yeh∗, Alexander G. Schwing.
(UIUC)

[XFeiF]

Motivation:
In existing semi-supervised learning methods, all unlabeled examples are equally weighted. The authors claim that not all unlabeled data are equal.

Main idea:
They propose an algorithm based on the influence function (a measure of a model's dependency on one training example) to assign a different weight for every unlabeled example. And the most important contribution they made is the fast and effective approximation of the influence function.

Some key information about the paper:

1. They automatically adjust the per-example weights by utilizing the influence function which estimates the "importance" of each unlabeled example using the validation set performance.
2. Two key problems when adjusting per-example weight: (a) Computation of per-example gradients; and (b) Computation of the inverse Hessian. To solve these two problems, the authors provide a theoretical analysis. (But it is hard to understand for me right now... )
3. In experiments part, they first analyze the effectiveness of their method on low-dimensional datasets before evaluating on standard SSL benchmarks including CIFAR10, SVHN, and IMDb. Finally, they ablate different components of the method to illustrate robustness and efficiency.