Mixed bandit & non-bandit fails with InvalidConfigurationError

ekalosak commented 4 years ago

A mixed input space configured exactly as in (https://github.com/SheffieldML/GPyOpt/blob/master/GPyOpt/core/task/space.py#L22) fails with an InvalidConfigError from (https://github.com/SheffieldML/GPyOpt/blob/master/GPyOpt/core/task/space.py#L188).

ekalosak commented 4 years ago

Solution appears to be simply deleting that InvalidConfigError block. Everything is working for me in preliminary testing using mixed bandit and non-bandit inputs.

zhenwendai commented 4 years ago

As far as I remember, this is an intentional choice, because the rest of the codebase has not been developed to handle a mix between bandit and non-bandit space. It may work in some cases but there are just no garantee about the behaviors of surrogate models and acquisition functions.

zhenwendai commented 4 years ago

If you have a strong use case, you are welcome to provide some testcases so that we can know whether it actually performs correctly.

javiergonzalezh commented 4 years ago

That's correct, indeed we removed that when we built Emukit to avoid those issues (and since the cases of application where minimal).

El vie., 27 mar. 2020 a las 14:51, Zhenwen Dai (notifications@github.com) escribió:

As far as I remember, this is an intentional choice, because the rest of the codebase has not been developed to handle a mix between bandit and non-bandit space. It may work in some cases but there are just no garantee about the behaviors of surrogate models and acquisition functions.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SheffieldML/GPyOpt/issues/310#issuecomment-605042718, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUJF46QJC4UAIZW5OFQ6DDRJS4QRANCNFSM4LUVYFKQ .

ekalosak commented 4 years ago

It's definitely a desirable functionality - it's actually a blocker for switching to Emukit at the moment. @javiergonzalezh to work this into Emukit, what would you suggest for an implementation strategy? Here are some ideas, obviously non-exhaustive:

Doing something like feeding low scores on violation of the domain is hamfisted and probably very wrong.
Adding a rejection criteria (like how constraints are implemented here) will be very slow, considering the relative cardinality of the bandit type domain and the product space of its component dimensions.
Exhaustively evaluate each bandit point in the acquisition optimization phase e.g. by conditionally optimizing the non-bandit quotient space. This seems more correct, but potentially ugly to implement and definitely slow.
Introduce an annealing or Metropolis-Hastings-like acquisition optimizer - there must be some good research on this already...

@zhenwendai I'm not sure I understand fully - are you aware of any specific spots in the codebase where this mixed bandit/non-bandit input space is a problem? Or is your recommendation out of an abundance of caution? I'll definitely keep this issue updated with concrete results as I get them. It seems like the surrogate models should be perfectly fine - the hyperparameter optimization isn't conditioned on input space restrictions. The acquisition optimizer definitely seems like it'd be an issue, and I'm sure my preliminary concerns listed above are non-exhaustive. Any insight you have on this issue would be very helpful.

ekalosak commented 4 years ago

Here's a PR supporting the mixed bandit / non-bandit input space: https://github.com/SheffieldML/GPyOpt/pull/313

SheffieldML / GPyOpt

Mixed bandit & non-bandit fails with InvalidConfigurationError #310