fmin-interface for bohb/cave

[shukon]

write an f-min interface as in scipy or smac to run BOHB (and subsequently CAVE, but that is trivial to add) with minimal configuration "out of the box". Optional parameters to finetune BOHB like budgets or parallelization.

[shukon]

This looks like a great start to me

I see the following issues in the current version:

• [ ] the example should include the budget in a meaningful way (that is not necessarily computationally expensive, i.e. number of datapoints used? or maybe use a different example?)
• [ ] func_args should be optional, as it does not contain parameters we optimize. We might have a function that already includes everything necessary.
• [ ] one of the main reasons we want f_min is to write configspace to disk during optimization process, so we need to do that
• [ ] runs with anything below maximum budget should be treated as estimates. This means, we want to return the best incumbent on the maximum budget, since better incumbents on lower budgets are, if anything, indicators for a weak budget-definition. (Our neural network example might be confusing, since there we actually do mix up a hyperparameter and a budget, so lower-budget runs are valid endresults...)
• [ ] (optional) calling the script (and triggering the main==name-procedure) should not invoke the example, but instead we should be able to directly pass e.g. a python-script (as a func) and a configspace and optimize on that. (python EasyOpt.py --func random_function.py --configspace cs.pcs)
• [ ] we want to generate a CAVE-report after optimization
• [ ] I'd personally rename store_path to output_dir

@mlindauer can you maybe check whether these points are actually what we talked about or which points need clarification?

[mlindauer]

This looks like a great start to me

To what do you refer here? I think that I don't understand some of your points because (for example) I don't know which example you are talking about. The jupyter notebook is not using such a simple interface so far; for example, the user should never see stuff such as workers and servers.

we want to generate a CAVE-report after optimization

the fmin interface does not need to generate the report directly. but it should be fairly easy to generate the report as a next step.

[shukon]

I think that I don't understand some of your points because (for example) I don't know which example you are talking about. The jupyter notebook is not using such a simple interface so far; for example, the user should never see stuff such as workers and servers.

My bad. I'm refering to the EasyOpt-script, that Philipp added.

[mlindauer]

Thanks. Yes, it looks like a good first version.

@shukon After a short look, I think I agree with all your points.

One more point: I would like to also have an argument such that users can define the number of parallel local workers to run.

[PhMueller]

Thanks for the feedback. I will include your ideas!

I hadn't had a good idea for a short example. But i will think about a better one, which uses the budget.

Did I understand this correctly: You, @mlindauer, don't want to generate a cave report in the script? Which is in my opinion the better way, because mixing or overloading the functionality may be confusing.

@shukon: I'm afraid, i currently don't have an idea how to accomplish the task, passing a file as command line argument and then calling the function from this file. But I will do some google-search on it. Or have you already done something like this before?

[mlindauer]

You, @mlindauer, don't want to generate a cave report in the script?

Yes, but it should be simple to generate the report after using the fmin interface.