microsoft / MLOS

MLOS is a project to enable autotuning for systems.
https://microsoft.github.io/MLOS
MIT License
138 stars 66 forks source link

mlos_webui design requirements #838

Open bpkroth opened 2 months ago

bpkroth commented 2 months ago

For starters, mlos_webui seems an appropriate moniker to me. It should support

  1. [ ] Visualization of ExperimentData from a shared Storage location.
  2. [ ] Execution of existing mlos_bench configs it has access to. (e.g., via SP access to a git repo).
    • [ ] We should allow it to be parallelized across Experiments via GridSearch (e.g., to schedule different instances of a tuning Experiment across different scenarios, like storage types or VM sizes, or benchmark type).
    • [ ] Error logs should be visible.
    • [ ] Experiments should be resumable.
  3. [ ] Editing of such configs. (with versioning)

_Originally posted by @bpkroth in https://github.com/microsoft/MLOS/pull/824#discussion_r1714331354_

bpkroth commented 2 months ago

@eujing @yshady

bpkroth commented 2 months ago

See Also #732

yshady commented 2 months ago

Thanks for compiling a list of tasks/features @bpkroth

I’d also love support for users to be able to

Essentially if we can map out everything we do into a web ui with no code I think this will be a far better costumer experience for those who may want to create their own benchmarks and their own tuning teams

yshady commented 2 months ago

I think creating a validation system where controls and optimized configs are done automatically on some level will be super valuable for a webpage

instead on launching many experiments manually we might want to create a system to find and validate a number of configs given a estimated monetary budget

along with this we also need to consider detection of when param selection is ineffective in changing a target metric