search

brain-research / realistic-ssl-evaluation

Open source release of the evaluation benchmark suite described in "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms"
Apache License 2.0
458 stars 98 forks source link

Randomness in build_tfrecords.py + label maps #17

Closed michalzajac-ml closed 5 years ago

michalzajac-ml commented 5 years ago

Hi @avital , first of all thanks for releasing the code, I find it very elegant, and tmuxp sessions for every experiment are really cool!

There is information in the README that in order to have exactly the same labeled/unlabeled divisions as in the paper, one has to use label maps from the repo. But the issue is, in order to run the experiments one has to first build tfrecords, which uses random shuffle. This means we HAVE TO create new label maps for these newly created tfrecords. Am I missing something, or there is really an issue here? How can we generate exactly the same tfrecords as in your experiments?

xgastaldi commented 5 years ago

Same question here

craffel commented 5 years ago

Thanks for pointing this out. We will investigate whether it's possible to get the desired behavior from appropriate settings of random seed.