Customized trial

[liuzhe-lz]

Current Design

User Side:

• Web UI adds a "clone" button for each trial
• Web UI adds a "tag" column in the trial details table (and maybe "edit tag" buttons)
• When user wants to create a customized trial, they click "clone" and a form pops up
• User edits the parameters and the tag (defaults to CUSTOMIZED maybe)
• User clicks "submit" button
• Web UI checks whether the parameters are out of search space
  • If yes, web UI warns the user that their parameter may break the tuner
• Web UI submits the parameters via REST API
• Maybe web UI should show a dialog saying the hyper-parameter is successfully submitted but may take some time to get scheduled and to show up
• If user wants to find the customized trial, they search its tag

Tuner Side

By default tuner algorithms will not receive metrics from customized trials. If an algorithm explicitly accepts customized trials (by implementing receive_customized_trial_result or by calling accept_customized_trials or whatever), results of all customized hyper-parameters will be reported to the tuner. NNI SDK does NOT guarantee the legality of customized hyper-parameters. Tuners are encouraged to check it themselves. However, since the user must confirm they know what they are doing before submitting an out-of-range hyper-parameter, tuners can feel free to ignore such case.

NNI Manager

TODO: check the current logic for appending jobs after experiment is done.

[liuzhe-lz]

Already implemented and released