Guarantee that sampled hyperparameters are the same for future commits

facebookresearch / DomainBed

DomainBed is a suite to test domain generalization algorithms

MIT License

1.42k stars 298 forks source link

Guarantee that sampled hyperparameters are the same for future commits #16

Closed SirRob1997 closed 4 years ago

SirRob1997 commented 4 years ago

Just opening this here to keep track of the issue mentioned in #12:

"We will have to find a solution to guarantee that the same random parameters are sampled in hparams_registry given the same random seed, for all future commits. Right now, adding algorithms or conditional logic in hparams_registry changes the random hyperparameters sampled for a given algorithm, dataset, and seed."

igul222 commented 4 years ago

Took a shot at fixing this. Let me know if you have thoughts.

SirRob1997 commented 4 years ago

Once this has been fixed it is probably a good idea to run a new sweep. From that point onwards adding a new algorithm should only require running the new algorithm on all datasets since sampled hparams stay consistent for the others.