mlos_bench: parameter constraints

[bpkroth]

For historical reasons, there are currently TunableGroups and Tunables in mlos_bench (the benchmarking portion) which are converted to ConfigSpace HyperParameters for mlos_core (the optimizer portion). Those are further converted to the appropriate backend (e.g., ConfigSpace for SMAC, or another variant of hyperparameters for FLAML).

ConfigSpace supports Conditions and ForbiddenClauses that could used to implement parameter constraints, but we haven't exposed it yet.

We have also off and on discussed replacing TunableGroups with ConfigSpace throughout, but that may also limit some of our ability to make other changes, so we've skipped it for now.

As a workaround for the moment one thing that could be done is a user driven "validation" script that evaluates the config suggested and returns a "bad" score immediately instead of executing the test.

Short of that, each optimizer's acquisition function needs to be adjusted to support this in the search space enumeration.

Related: https://github.com/microsoft/MLOS/issues/403

[bpkroth]

Some things to note:

• TunableGroups provide reusable parameter specifications for individual Environments (e.g., VM, OS, App)
• They can be combined in an Experiment to do cross Environment co-tuning.
• Constraints may similarly need to be specified both within a TunableGroup and across TunableGroups. For instance:
  • app.min_file_size < app.max_file_size
  • app.buff_size_mb < (vm.ram_size_gb - 1) * 1024

This will require some care in thinking through what the config specification looks like as well as how to handle it in either TunableGroups vs ConfigSpace conversions or else automatic enforcement in the mlos_bench.Optimizer side of things.

[poojanilangekar]

Hi @bpkroth,

I had a look at the Conditions and the ForbiddenCluases. However, I am unsure how to implement conditions between hyperparameters. I think the conditions work on the child hyperparameter based on the parent hyperparameter and value. On the other hand, the forbidden clauses work on a hyperparameter/value(s) pair. From what I understand, we can't directly add a condition between two hyperparameters, is this correct or am I missing something here?

Thanks, Pooja

[bpkroth]

I haven't thought through the full implementation in mlos_bench yet, but it likely starts with something here:

• https://github.com/microsoft/MLOS/tree/main/mlos_bench/mlos_bench/config/schemas/tunables
• https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/tunables/tunable_groups.py
• https://github.com/microsoft/MLOS/blob/main/mlos_bench/mlos_bench/optimizers/convert_configspace.py

[bpkroth]

Regarding your observation, I think it's mostly a documentation issue. This PR in ConfigSpace seems to indicate that Forbidden clauses between Hyperparameters, not just constants, is possible.

https://github.com/automl/ConfigSpace/pull/245/files

How I might handle this is something like the following:

• a separate config structure to define conditions and forbidden clauses
  • I can probably help with the json schema when we get to it, but I think something like the following to start with might be OK:

    {
    "constraints": [
      {
        // "op_name": ["param_name_1", "param_name_2"] --> param_name_1 {op_name} param_name_2
        "less_than": ["param_a", "param_b"]
      },
      {
        // Would have to be converted to a reverse "ForbiddenGreaterThanRelation" internally
        "less_than_equal": ["param_c", "param_d"],
      },
      {
        "equal": ["param_x", "param_y"],
      },
      ...
    ]
    }

• allow them to span different TunableGroups definitions (e.g., VM Environment TunableGroups and App Environment TunableGroups)
• when converting from TunableGroups to ConfigSpace, apply the Forbidden constraints only for Tunables that exist in the set (e.g., if a Condition exists declaring vm_ram > buffer_pool_size, but vm_ram is not in the included TunableGroups, issue a Warning but omit it)

Since not all backend optimizers support ConfigSpace (e.g., FLAML) we might also need to implement an internal "rejection" method inside mlos_bench for this (e.g., automatically report a bad value back when a suggested value returned by the optimizer is invalid)

[bpkroth]

@motus any other thoughts?

[bpkroth]

Related: https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function/#more-constraints-on-the-tuning

[poojanilangekar]

Thanks for getting back to me about this, I will go over the linked documentation/PRs and reach out once I have a better idea of the approach.

[bpkroth]

FWIW: https://github.com/automl/ConfigSpace/issues/352

[eddiebergman]

Hi @poojanilangekar, I am maintaining ConfigSpace and hopefully we can get towards making this possible :)