igul222
commented
4 years ago I'm hesitant to support this case as a built-in for two reasons:
- the meaning of reported errors won't be consistent across papers (all else equal, more standardization is good)
- I believe that ignoring variability in the hparam selection process can lead to misleadingly good results, e.g. if you run the sweep multiple times and only report the best number. In theory you could get arbitrarily good numbers (and low stdev) this way, but they wouldn't be reproducible.
There's no easy way to see hparams corresponding to the best run, but a starting point for writing this functionality yourself would be here. You want to change this to something like map(lambda _, run_records: (self.run_acc(run_records), run_records[0]["hparams_seed"]) ) and then propagate that hparams_seed all the way to the final results table.
SirRob1997
As it is currently implemented, launching a sweep with
n_trials> 1 allows for reporting std. deviation of the best performing hyperparameter setting based on the different validation techniques. If I correctly understood the implementation, this computesn_trialsdifferent in- and out-splits and runs them for each of the sampled hyperparameter settings.Users with limited resources who are implementing a new algorithm might not want to do that. A more convenient and faster way would be an option where users are able to tune hyperparameters and then run ONLY the best performing setting for each of the validation techniques on additional
n_trialsseeds to report std. deviations of choosing different in- and out-splits.Also: What is the current way of obtaining the best performing hyperparameter setting for each of the validation techniques?