search

microsoft / MLOS

MLOS is a project to enable autotuning for systems.
https://microsoft.github.io/MLOS
MIT License
139 stars 66 forks source link

Ability to schedule ablation study of optimized configs #860

Open bpkroth opened 1 month ago

bpkroth commented 1 month ago

Related to #807 it would also be useful to be able to schedule an ablation study of optimal configs vs. the defaults by toggling single parameters (or combinations) back to the default in order to determine what improved in the optimal config.

bpkroth commented 1 month ago

@eujing

bpkroth commented 1 month ago

@jsfreischuetz

bpkroth commented 1 month ago

See Also: #687

eujing commented 1 month ago

I think this can also be achieved with the manual optimizer in #855, if we list out the ablation configs explicitly as configs to execute during the experiment. I guess the advantage is that we get repeats of each config in a cycle if we did it that way.

bpkroth commented 1 month ago

I think the whole point is to automate the enumeration of the configs. I'd rather see us implement #687 and then extend it by adding a shedule_ablation_study(config) for a given config vs. the defaults. Then any scheduler can determine which one to run first (or in parallel) - it doesn't really matter.

To use @motus terminology from a previous discussion, this is a method of enqueuing Trials into the queue (e.g., config suggestion), not of scheduling within that set.