search

mlr-org / ParamHelpers

Helpers for parameters in black-box optimization, tuning and machine learning.
https://paramhelpers.mlr-org.com
Other
25 stars 9 forks source link

Description of tunable is misleading #198

Closed jakob-r closed 5 years ago

jakob-r commented 5 years ago

@zkurtz noted in this mlrMBO issue that the description if the tunable flag is misleading. In fact it does not only affect mlrMBO but also tune tuning methods im mlr and probably others, because they do not respect that flag.

At the moment it reads:

Is this parameter tunable? Defining a parameter to be not-tunable allows to mark arguments like, e.g., “verbose” or other purely technical stuff, and allows them to be excluded from later automatic optimization procedures that would try to consider all available parameters. Default is TRUE (except for untyped, function, character and characterVector) which means it is tunable.

My suggestions:

Is this parameter tunable? Defining a parameter to be not-tunable allows to mark arguments like, e.g., “verbose” or other purely technical stuff. Note that this flag is most likely not respected by optimizing procedures unless stated otherwise. Default is TRUE (except for untyped, function, character and characterVector) which means it is tunable.

mb706 commented 5 years ago

Maybe the tunable flag should only exist for LearnerParam where declaring tunability in metadata makes sense.

berndbischl commented 5 years ago

because they do not respect that flag.

yes, i know. that was kinda by intention. but it is still bad.

Is this parameter tunable? Defining a parameter to be not-tunable allows to mark arguments like, e.g., “verbose” or other purely technical stuff. Note that this flag is most likely not respected by optimizing procedures unless stated otherwise. Default is TRUE (except for untyped, function, character and characterVector) which means it is tunable.

this is surely an improvement of the docs.

IMHO: tunable should be removed (and here i expecially mean: not in included in paradox). it is very hard to define what is tunable and what is not. and it the user (or automatic algorithms) define anyway what they want to tune. or learn this.

jakob-r commented 5 years ago

@berndbischl It is clear, that it wont make it into paradox.

It sounds logical what @mb706 is saying but I don't want to change the API anymore. Who knows what this would break in others people code?

Therefore i just updated the docs. @berndbischl merge?

berndbischl commented 5 years ago

Totally agree, do not change API here, but clearly and correctly document the current state. New stuff we should improve in paradox. --->Merge