Hyper-parameter overview top trials order

[TimSchim]

In the WebUI under: Trial details -> Hyper-parameter one can choose to show the top xx% trials. By default this shows the trials with the highest scores. Is there a way to change this to showing the trials with the lowest scores, e.g. if my metric is MSE? So far my workaround is to just to multiply my score by (-1).

[QuanluZhang]

@Lijiaoa please help follow this issue.

[Lijiaoa]

@TimSchim Why not set optimize_mode to minimize in config? If you set it to minimize, webui will filter best trials as small final metric.

[TimSchim]

This seems to only work when using a tuner that uses this option. When I use Random instead of TPE I get TypeError: __init__() got an unexpected keyword argument 'optimize_mode'

Tried these two options:

• experiment.config.tuner.class_args = {'optimize_mode': 'minimize'}

• experiment.config.tuner.class_args['optimize_mode'] = 'minimize'

when setting the config from Python like here.

[Lijiaoa]

It seems that random tuner doesn't support this arg by doc.

[TimSchim]

Any other suggestions or should this be a new feature for future versions of the Web Portal?

[Lijiaoa]

You could change this line into convertedTrials.sort((a, b) => (a[k] - b[k])); and delete the EXPERIMENT from this line. And you could follow the Readme to format the code.

@TimSchim this issue had been fixed in PR #4850.

[ThePhoenixCoding]

I'm having the same problem and can add some information: I use RegularizedEvolution which supports optimization_mode = 'minimize'. The Hyper Parameters Section in the UI correctly colors the best (lowest) and worst (highest) values, but the "Top x%" still filters out (= removes) the lowest values and thus the best models. It does not care about the optimization mode.

[Lijiaoa]

@ThePhoenixCoding Could you paste your config.yml in here? Because webui does deal with the best trials with Top x%.

[ThePhoenixCoding]

This one?

{ "params": { "trialCommand": "python3 -m nni.retiarii.trial_entry py", "trialCodeDirectory": "C:\Users\alexa\Desktop\Code\NAS_New_Trial", "trialConcurrency": 1, "trialGpuNumber": 1, "maxTrialNumber": 5, "debug": false, "experimentWorkingDirectory": "C:\Users\alexa\nni-experiments", "trainingService": { "platform": "local", "trialCommand": "python3 -m nni.retiarii.trial_entry py", "trialCodeDirectory": "C:\Users\alexa\Desktop\Code\NAS_New_Trial", "trialGpuNumber": 1, "debug": false, "useActiveGpu": true, "maxTrialNumberPerGpu": 1, "reuseMode": false }, "executionEngine": "py" }, "execDuration": "1m 23s", "nextSequenceId": 3, "revision": 13 }

[Lijiaoa]

so actually you didn't set optimization_mode = 'minimize' in your config. If you set it in your config.yml file you could get right Top x% usage.

[ThePhoenixCoding]

Thanks! Nonetheless this seems to be very unintuitive for me. I already set the optimization mode in the search strategy (and in export_top_models afterwards). It would be more user friendly if that would also carry to the UI.

[QuanluZhang]

cc @ultmaster for awareness