search

brain-research / realistic-ssl-evaluation

Open source release of the evaluation benchmark suite described in "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms"
Apache License 2.0
458 stars 98 forks source link

add an argument to use only labeled data in training process. #7

Closed DoctorKey closed 5 years ago

DoctorKey commented 5 years ago

Maybe, it is a simple change that can resolve the problem mentioned by #6 .

avital commented 5 years ago

This looks probably good, but one thing: this is changing the effective batch size for the fully supervised experiments to be twice as large as original code.

Because batch size can affect the results of experiments, I think we should explicitly make the batch size half the size in the .yml files.

What do you think?

DoctorKey commented 5 years ago

BN layers use 100 batch size in original code. Just making the batch size half will change the computation of BN layers, although it maintains the effective batch size for loss. Maybe, there is a need to optimize the hyperparameters again.

avital commented 5 years ago

Here's what I propose we do:

Run all of the fully-supervised baselines with this pull request, in two ways: with batch size 100 and batch size 50. Hopefully the results end up being very close to what we have reported in the paper.

In case batch size of 50 or 100 matters, or if the results end up much different than our original published results, then we may want to re-tune the hyperparameters, which will take longer.

What do you think?

DoctorKey commented 5 years ago

OK. Good luck to you.

avital commented 5 years ago

Hi @DoctorKey, thanks again for the PR. I made some comments, please take a look. Also, have you run this code? Can you report the results for CIFAR-10 fully supervised and SVHN fully-supervised before and after this change? (the table-1 runs).

JiechengZhao commented 5 years ago

@DoctorKey Thanks for the code, we are also wondering the effect that unlabeled data acts on BN layer. We use your code to have a test on SVHN. there is merely no change in the result. But we only run parts of the experiment for the limited resources.

DoctorKey commented 5 years ago

I run the code when I made this PR. Because of the limited resources, I only got the result of CIFAR-10 fully supervised. We found that the result was almost unchanged.

diego898 commented 5 years ago

hey @avital - just a friendly ping to see if this PR (and repo) are still planned to be merged. Thanks!

avital commented 5 years ago

Merged, thanks for the ping.