search

brain-research / realistic-ssl-evaluation

Open source release of the evaluation benchmark suite described in "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms"
Apache License 2.0
460 stars 98 forks source link

Add run for ImageNet pre-training with fewer classes #4

Closed avital closed 6 years ago

avital commented 6 years ago

This lets us train only on the subset of ImageNet classes that don't overlap with CIFAR10 classes.

avital commented 6 years ago

(I am starting the training run now)

craffel commented 6 years ago

The class list matches what I have locally, without horse-cart as intended. Commands look good, minor nit that it may be a little clearer to not set the --consistency_model flag because we are not using SSL here (it will be set to mean_teacher by default, but its value doesn't actually matter since max_consistency_coefficient=0.

craffel commented 6 years ago

(will approve once we confirm it trains and something reasonable happens)

avital commented 6 years ago

Training accuracy ended up at 56.3%, compared to 54.6% for the normal unfiltered run. (I didn't run eval on the non-filtered run recently so I can't compare those). This is expected -- we have fewer training examples so with the same regularization one would expect slightly higher training accuracy.

I'm starting the fine-tuning run now.

avital commented 6 years ago

We have results on "filtered ImageNet without CIFAR classes -> CIFAR" finetuning. It gets 12.91% test error on best validation error

Compared to 12.0% as we report in the paper on "Full ImageNet -> CIFAR" fine-tuning It's still better than all of the SSL techniques on the same model.

So we should probably report both and explain. And we can consider adding the confusion matrix experiment, or not. I am open to both.