search

brain-research / realistic-ssl-evaluation

Open source release of the evaluation benchmark suite described in "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms"
Apache License 2.0
458 stars 98 forks source link

How is the error rate and uncertainty calculated? #29

Closed varunnair18 closed 5 years ago

varunnair18 commented 5 years ago

Hello,

From reading the paper, it seems that the error rates are calculated from the test error chosen at the point of lowest validation error, is that correct?

Also, do you perform experiments with multiple random seeds, or is uncertainty calculated using several different validation sets? How exactly was that uncertainty calculated? Forgive me if I missed it as a detail in your paper.

Thank you!

craffel commented 5 years ago

Hi,

From reading the paper, it seems that the error rates are calculated from the test error chosen at the point of lowest validation error, is that correct?

Yes.

do you perform experiments with multiple random seeds

Yes.

is uncertainty calculated using several different validation sets?

The same validation/text set is used in each experiment.