search

automl / CAVE

[deprecated] Configuration Assessment, Visualization and Evaluation
https://www.automl.org
BSD 3-Clause "New" or "Revised" License
45 stars 13 forks source link

Evaluating the default configuration in BOHB #222

Open ashraaghav opened 5 years ago

ashraaghav commented 5 years ago

I tried using CAVE with BOHB and I noticed that the default configuration is not used in the report at all, unlike in SMAC3. I understand it is because BOHB does not necessarily evaluate the default configuration, but it will be nice to compare the incumbent performance's with the default's in CAVE, especially in the configurator footprint plot. Is it possible in CAVE?

mlindauer commented 5 years ago

Hi,

Thank you for your feedback. Unfortunately, so far it is not possible because CAVE does not evaluates any configuration but it only uses the data provided to it. We could use a performance estimate from an EPM; but since BOHB does not use the default configuration anyway, there is no point in comparing the default against BOHB's incumbent. Maybe if we could force BOHB (in the future) to consider a given default configuration, we could do a similar treatment like we do with SMAC3.

Overall, I see where you are coming from. We will discuss this internally again, but it is unlikely that we will provide a solution in the near future.

Best, Marius