Nested hyperparameters cannot be updated

[dpaetzel]

Describe the bug

I want to optimize the hyperparameters of a simple pipeline. According to the MLJ docs, this should be possible by accessing these hyperparemeters using :(dot.syntax.like.this) (which works, e.g., for Grid tuning). However, this does not work with MLJTreeParzenTuning.

To Reproduce

An MWE is

using TreeParzen
using MLJ

standardizer = Standardizer()
DT = @load DecisionTreeRegressor pkg = DecisionTree
model = DT()
pipe = Pipeline(standardizer = standardizer, model = model)

X, y = make_regression(100, 5)

function usingexpr()
    space = Dict(:(model.max_depth) => HP.Choice(:(model.max_depth), collect(1:10)))
    pipe_tuned = TunedModel(
        model = pipe,
        tuning = MLJTreeParzenTuning(),
        range = space,
        measure = mae,
    )
    mach_tuned = machine(pipe_tuned, X, y)
    fit!(mach_tuned)
end

function usingsymbol()
    space = Dict(
        Symbol("model.max_depth") =>
            HP.Choice(Symbol("model.max_depth"), collect(1:10)),
    )
    pipe_tuned = TunedModel(
        model = pipe,
        tuning = MLJTreeParzenTuning(),
        range = space,
        measure = mae,
    )
    mach_tuned = machine(pipe_tuned, X, y)
    fit!(mach_tuned)
end

function usinggrid()
    space = range(pipe, :(model.max_depth); lower = 1, upper = 10)
    pipe_tuned = TunedModel(model = pipe, tuning = Grid(), range = space, measure = mae)
    mach_tuned = machine(pipe_tuned, X, y)
    fit!(mach_tuned)
end

# usingexpr()
# ERROR: LoadError: MethodError: no method matching TreeParzen.HP.Choice(::Expr, ::Vector{Int64})
#
# Closest candidates are:
#   TreeParzen.HP.Choice(::Symbol, ::Vector)
#    @ TreeParzen ~/.julia/packages/TreeParzen/gdoI0/src/HP.jl:116

# usingsymbol()
# ERROR: LoadError: Invalid hyperparameter: model.max_depth
# Stacktrace:
#   [1] error(s::String)
#     @ Base ./error.jl:35
#   [2] update_param!(model::MLJBase.DeterministicPipeline{NamedTuple{(:standardizer, :model), Tuple{Unsupervised, Deterministic}}, MLJModelInterface.predict}, param::Symbol, val::Int64)

usinggrid()
# Works.

Expected behavior

I expected to be able to tune nested hyperparameters just like when using the Grid strategy.

Environment (please complete the following information):

• OS: Linux
• Version: NixOS at github:dpaetzel/nixpkgs/dpaetzel/nixos-config .
• Julia Version: 1.9.3
• TreeParzen.jl Version: 0.3.3

Additional context

Thank you for making and maintaining this library! :slightly_smiling_face:

It looks like the problem is caused by the update_param! function in MLJTreeParzen.jl not respecting nested hyperparameters.

[dpaetzel]

I'm currently looking into this. I think one could (and should?) reuse MLJBase.recursive_setproperty! but then the Choice (and other HP) constructor still would have to accept an ::Expr which it does not right now. Would you be interested in a PR?

[dpaetzel]

I think it may be possible to solve this by simply replacing Symbol with Union{Symbol, Expr}?

However, it would probably make sense to extract the hyperparameter label type (which is currently hardcoded to Symbol) to a definition such as

Label = Union{Symbol,Expr}

which can then be used in all the Dicts etc.

However, I'm not vastly familiar with the intricacies of Tree-structured Parzen Estimation (and neither with this implementation) and am a little bit worried that this would break stuff. Maybe a person more familiar with the code can comment? :slightly_smiling_face:

[kainkad]

Hi @dpaetzel thank you for raising this issue. I've checked your examples and the issue is not happening because of update_params! not supporting nested hyperparams but as you correctly observed the space with hyperparams to be optimised should be a dictionary where the keys (params) are Symbols and not Expressions or Symbols of those expressions. Given your example, the model param should be :max_depth rather than model.max_depth or Symbol("model.max_depth"). Based on your MWE, you created the pipeline object containing both the standardizer and the model and you passed that pipeline object to the model entry in the TunedModel() wrapper. This means that accessing the model params would then indeed require to be pipe.model.param but if instead you provided model=pipe.model then your space can be defined as below:

function usingexpr()
           space = Dict(
               :max_depth => HP.Choice(:max_depth, collect(1:10))
           )
           pipe_tuned = TunedModel(
               model = pipe.model,
               tuning = MLJTreeParzenTuning(),
               ranges = space,
               measure = mae,
           )
           mach_tuned = machine(pipe_tuned, X, y)
           fit!(mach_tuned)
       end

Would the above work for you?

If you’re interested in PR, then may be parsing the expression and extracting the hyperparm name to be passed to TreeParzen could be an option but this could be tricky if there were further changes to the MLJ's dot syntax.