Open CameronBieganek opened 9 months ago
The docs could probably specify that the model factory function is provided with an object whose properties are the same as the keyword arguments to Domain
. So, the user can use argument destructuring as described in the previous post, or they can access hyperparameters via getproperty
, like this:
domain = Domain(
K = (1, 3, 5, 7, 11),
leafsize = (5, 10, 15)
)
function make_model(params)
Pipeline(
encoder = ContinuousEncoder(),
estimator = KNNRegressor(K=params.K, leafsize=params.leafsize)
)
end
tunable_model = TunedModel(make_model, domain; strategy=Grid())
or like this:
tunable_model = (
TunedModel(domain; strategy=Grid()) do params
Pipeline(
encoder = ContinuousEncoder(),
estimator = KNNRegressor(K=params.K, leafsize=params.leafsize)
)
end
)
or this:
tunable_model = (
TunedModel(domain; strategy=Grid()) do params
(; K, leafsize) = params
Pipeline(
encoder = ContinuousEncoder(),
estimator = KNNRegressor(K=K, leafsize=leafsize)
)
end
)
or this:
tunable_model = (
TunedModel(domain; strategy=Grid()) do params
Pipeline(
encoder = ContinuousEncoder(),
estimator = KNNRegressor(; params...)
)
end
)
@CameronBieganek Thanks for pitching in with this idea. Here are a few drawbacks that come to mind. I wonder what you think about them:
Essentially you are replacing an inspectable, mutable hyper-parameter, of a TunedModel
with a function, which is neither inspectable, nor mutable, except by total replacement. This seems to limit composition. For an admittedly contrived example, suppose I want to tune hyper-parameters one at a time, with a sequence of wrappings, as in TunedModel(model=TunedModel(model=SomeAtomicModel(), range=r1), range=r2)
where r2
refers to (nested) hypeparameters of SomeAtomicModel()
. This would not be possible. Do you see that as an issue?
With the atomic model types hidden, I have a problem with forwarding traits of the AtomicModel
to the wrapper TunedModel(model=AtomicModel())
. For example, if I'm wrapping a probabilistic predictor model
(prediction_type(model) == :probabilistic
) as a TunedModel
, then the wrapped model must also be seen as probabilistic. How should that work here? (Currently, we also need to get an abstract model type as a type parameter for the wrapper, which would similarly be opaque in your suggested scheme.)
@ablaom I will try to respond to your questions soon, but for now I just want to point out a couple advantages of this approach.
The current hyperparameter tuning interface requires that the fields of model types be public API, otherwise the user wouldn't know the correct :foo
symbol to write. With my proposal above, those fields would no longer need to be public API. Of course the part of the API that is still public is the keyword argument constructor for a model. (But of course there is no requirement that the keyword argument names match the field names of the model type.)
Pipeline
no longer needs a keyword argument constructor:The only reason that Pipeline
needs a keyword argument constructor right now is to make it easier for users to specify the right expression for nested parameters in hyperparameter tuning, e.g.
:(random_forest.min_samples_leaf)
. With my proposal above this would no longer be necessary, so the positional argument Pipeline
constructor would be sufficient.
That might seem like a small win, but interface simplifications are always a win in my book, and this would bring MLJ closer to alignment with TableTransforms.jl.
I think I mentioned this idea a few years ago on a different thread, but I never gave the proposal its own issue.
This is a breaking change, but I think it would provide a cleaner interface for hyperparameter tuning, and it would also address #174.
The idea is that instead of specifying the hyperparameters that need to be tuned with a quoted expression like
:(estimator.leafsize)
, the user can simply provide a function that creates a new model. Here is one way that it could work:I've prefixed the feature names with an underscore to emphasize that what matters is that the keyword argument names in
make_model
match the keyword argument names inDomain
. The actual names are arbitrary.The downside to requiring the user defined function to use keyword arguments is that you can't really use
do
-notation inTunedModel
, because as far as I can tell there is no way to define an anonymous function with keyword arguments by usingdo
-notation. So, an alternative interface would be to require the user defined function to take a single positional argument with property destructuring, like this:This can then be expressed with
do
-notation as follows:I believe this interface is generic enough to work for any hyperparameter tuning strategy.