search

hyperopt / hyperopt

Distributed Asynchronous Hyperparameter Optimization in Python
http://hyperopt.github.io/hyperopt
Other
7.26k stars 1.06k forks source link

qlen computation in base.Trials.run #77

Closed yamins81 closed 12 years ago

yamins81 commented 12 years ago

I THINK that the follow statement should be true:

1) If you initialize an experiment to have async = False

2) if you're in a totally serial execution environment

3) If max_queue_len = 1

THEN: during an experiment run, there should be no circumstance in which suggest is called twice before serial_execute is called once.

However, currently this is not the case, because even if a job has just been suggested in the inner while loop of the run method, the line qlen = get_queue_len() can still return 0, before the serial_execute to run the job has been called.

I think this is a bug.

jaberg commented 12 years ago

Could you PR a unittest illustrating this?

yamins81 commented 12 years ago

Well, the sync bandit test in eccv does this

jaberg commented 12 years ago

Pushed a fix

jaberg commented 12 years ago

Looking beyond a month from now, it's good if unittests for hyperopt are actually in hyperopt, right?

yamins81 commented 12 years ago

Sure! and I'd be happy to write the test -- but I'm not sure what exactly was causing the problem (otherwise I'd have fixed it) , so I wasn't sure how hard it would be to actually construct the condition that was causing it, and the eccv unittest was itself not appropriate for hyperopt. Was it something specific about the bandit_algo? Or was it a general problem that cause qlen = 0 always at the beginning (so therefore it would have been trivial to test for?)

jaberg commented 12 years ago

Did you see the fix I pushed? On Feb 16, 2012 8:15 PM, "Dan Yamins" < reply@reply.github.com> wrote:

Sure! and I'd be happy to write the test -- but I'm not sure what exactly was causing the problem (otherwise I'd have fixed it) , so I wasn't sure how hard it would be to actually construct the condition that was causing it, and the eccv unittest was itself not appropriate for hyperopt. Was it something specific about the bandit_algo? Or was it a general problem that cause qlen = 0 always at the beginning (so therefore it would have been trivial to test for?)

Reply to this email directly or view it on GitHub: https://github.com/jaberg/hyperopt/issues/77#issuecomment-4013345