tune: support for conditional / nested / hierarchical parameter search spaces

[bbudescu]

This is not documented within the tutorial, so I assumed that, most likely, it's not directly supported. However, I see that there is some reference to hierarchical spaces in the api docs and a discussion about it in #717. I don't understand exactly the code samples there, as I focused only on the black box optimization part, and ignored the AutoML part, but I assume that in AutoML, if you need to choose an alg and then tune it, and every algorithm has specific parameters, than you kinda need nested search spaces.

Also, I think the constraints api can be used to mark incompatible configurations as invalid. I'm not sure, though, how that affects tuning performance. I haven't taken the time to properly understand the BlendSearch algorithm and how it trades off local with global search, but I imagine that the bayesian model underlying at least the global search (that must be a Gaussian Process, or something, right?), that serves as a predictor for the performance which would be attained by a certain config before running it, would have some sort of smoothness prior. Now, if using constraints assigns very low scores to invalid configurations, due to this smoothness prior, wouldn't it also assign low scores probs to the configs at the edge of feasibility? I imagine that there might be very good combinations right at the edge of the feasible region, but they will be explored later rather than sooner, and, although kinks can be modeled, they are still considered less likely than smooth regions.

So, in short, is it safe to use the relational constraints to implement conditional search spaces? Is there another, better way to let the optimizer know that a particular combo is not feasible?

Also, I know that some other black box optimization packages, e.g., HyperMapper train a separate classifier model for guessing whether a certain config is feasible or not. Do you guys use such a model to drive sampling, too?

I mean, it has the advantage that you don't have to specify the invalid space up front, which is nice if you don't know the invalid combinations a priori, and you want the optimizer to learn this by itself, but this is not very helpful if you know the invalid configs and they can be expressed through conditional rules, and want to inject this info into the model so it doesn't waste time trying out bad combinations.

So, can I define conditional search spaces and, if yes, how?

Also, It wouldn't hurt to make a reference in the tutorials about this, too, even if it's something like "we don't support this right now".

L.E.: now I can see that there's this is_valid method of Domain objects. Can (and should) the user override this function somehow to specify arbitrary valid combinations?

[sonichi]

"config constraints" can be used to specify the invalid combinations: https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function#more-constraints-on-the-tuning Hierarchical and nested search space are supported. We should document them. cc @qingyun-wu

[bbudescu]

Is there perhaps an example of how to use hierarchical / nested search spaces that I can use until you guys are done with the docs?

[sonichi]

Is there perhaps an example of how to use hierarchical / nested search spaces that I can use until you guys are done with the docs?

hierarchical: https://github.com/microsoft/FLAML/blob/5eb99276423f8ac88b7fe38c37d9b7f37fafe4fb/test/tune/test_space.py#L47 nested: https://github.com/microsoft/FLAML/blob/5eb99276423f8ac88b7fe38c37d9b7f37fafe4fb/test/tune/test_tune.py#L221

[bbudescu]

how about more advanced conditional search spaces than a plain choice (e.g., similar to #988)?