mlos_webui design requirements

[bpkroth]

For starters, mlos_webui seems an appropriate moniker to me. It should support

1. [ ] Visualization of ExperimentData from a shared Storage location.
2. [ ] Execution of existing mlos_bench configs it has access to. (e.g., via SP access to a git repo).
   • [ ] We should allow it to be parallelized across Experiments via GridSearch (e.g., to schedule different instances of a tuning Experiment across different scenarios, like storage types or VM sizes, or benchmark type).
   • [ ] Error logs should be visible.
   • [ ] Experiments should be resumable.
3. [ ] Editing of such configs. (with versioning)

• [ ] Authentication should be required.
• [ ] Fine grained authorization (e.g., restricting access to experiments by group or some such) is out of scope for now (i.e., if you can authenticate, you can perform any action).
• [ ] Code should be modular enough to allow easily adding features (e.g., new visualizations).

_Originally posted by @bpkroth in https://github.com/microsoft/MLOS/pull/824#discussion_r1714331354_

[bpkroth]

@eujing @yshady

[bpkroth]

See Also #732

[yshady]

Thanks for compiling a list of tasks/features @bpkroth

I’d also love support for users to be able to

• be able to configure benchmarks from a gui
• Error logs should show the end x lines and parse out potential error lines (user preference)
• be able to monitor active experiments and see which launched experiments have quit before hitting the needed trial repeat number, this can be used as a tracking method
• automated correlation detection / validation between Params and a target metric would be a heaven sent
• for a experiment grid showing pie chart for each part where user can see pending, failed, and success metrics

Essentially if we can map out everything we do into a web ui with no code I think this will be a far better costumer experience for those who may want to create their own benchmarks and their own tuning teams

[yshady]

I think creating a validation system where controls and optimized configs are done automatically on some level will be super valuable for a webpage

instead on launching many experiments manually we might want to create a system to find and validate a number of configs given a estimated monetary budget

along with this we also need to consider detection of when param selection is ineffective in changing a target metric