mlr-org / ParamHelpers

Helpers for parameters in black-box optimization, tuning and machine learning.
https://paramhelpers.mlr-org.com
Other
26 stars 9 forks source link

add 'distinct' argument to VectorParams #206

Closed pat-s closed 5 years ago

pat-s commented 5 years ago

To enable sampling without replacement.

Currently, sampling with replacement is hardcoded for vector params.

This is needed to sample unique filter methods which are going to be used in ensemble filters.

@jakob-r Is this possible in paradox already?

Edit: The fix obv only works for tuneXXX methods that use sampleValue. MBO does not. So maybe we need to tackle this in multiple repos..

jakob-r commented 5 years ago

I don't know if we want to to really deal with that here. It will introduce more problems:

As we already discussed we can transform a logical vector param to a distinct factor vector. Also is this just a restriction which you could also include in the forbidden argument of the param forbidden = quote(anyDuplicated(id.of.numvec)) should do the trick.

I vote for:

berndbischl commented 5 years ago

i also dont think we should do this here. but can we please explain the "issue" first? either in an issue here, or in paradox?

pat-s commented 5 years ago

but can we please explain the "issue" first?

Quoted from above:

"Currently, sampling with replacement is hardcoded for vector params. This is needed to sample unique filter methods which are going to be used in ensemble filters."

Addition: If you create a "DistinctVectorParam" with let's say 15 values and set len = 8, 8 instances will be sampled WITH replacement out of those 15 choices. This is hardcoded atm and in my use case, I need sampling without replacement as I do not want to have multiple instances of the same simple filter in this selection.

Should I open an issue in paradox?

berndbischl commented 5 years ago

i read the code, i also read your text here, still had some problems to understand what exactly is to be adressed.

Should I open an issue in paradox?

yes, i think it makes sense to continue discussing there. i have some things to mention

berndbischl commented 5 years ago

let me try to summarize here already:

a) the current PH implementation allows only for uniform sampling. you now kinda change that, just for a special case.

b) IMHO we are looking at a design problem. params should not implement their sampler. they should simply descibe params. the sampler should be a separate service class. then this can be specialized