"applicable method may be too new" error instantiating models loaded in function scope with `@load`

[vishalhedgevantage]

When I include a script from the terminal which contains the code to load RandomClassifer from DecisionTree package, I get the following error -

ERROR: LoadError: MethodError: no method matching MLJDecisionTreeInterface.RandomForestClassifier() The applicable method may be too new: running in world age 30054, while current world is 30055. Closest candidates are: MLJDecisionTreeInterface.RandomForestClassifier(; max_depth, min_samples_leaf, min_samples_split, min_purity_increase, n_subfeatures, n_trees, sampling_fraction, pdf_smoothing) at none:0 (method too new to be called from this world context.)

The specific line in the code where it is complaining is - tree = (@MLJ.load RandomForestClassifier pkg=DecisionTree verbosity=0)()

I also saw that once I call this code on its own in the terminal then things work fine.

[ablaom]

Thanks for reporting. Based on what you have provided I cannot say what this error can be. Can you please provide a minimum working example that will allow me to reproduce?

Please also report the output of Pkg.status() and versioning().

[vishalhedgevantage]

This is how I can simulate this error - Create a script with the following lines. Start a terminal and activate the environment and make sure that RandomForest has not been loaded prior to running this script. In the terminal window, include this script.

using MLJ, DecisionTree

function test() (@MLJ.load RandomForestClassifier pkg=DecisionTree verbosity=0)() end

test()

This error goes away by making sure that RandomForestClassifier is loaded prior to function call. Here is the script which doesnt result in error

using MLJ, DecisionTree

function test() (@MLJ.load RandomForestClassifier pkg=DecisionTree verbosity=0)() end

(@MLJ.load RandomForestClassifier pkg=DecisionTree verbosity=0)() test()

The output of Pkg.status() is

[336ed68f] CSV v0.8.5 [324d7699] CategoricalArrays v0.10.0 [a93c6f00] DataFrames v1.1.1 [7806a523] DecisionTree v0.10.10 [c91e804a] Gadfly v1.3.3 [f67ccb44] HDF5 v0.15.5 [7073ff75] IJulia v1.23.2 [4138dd39] JLD v0.12.3 [23992714] MAT v0.10.1 [add582a8] MLJ v0.16.6 [c6f25543] MLJDecisionTreeInterface v0.1.2 [91a5bcdd] Plots v1.16.5 [438e738f] PyCall v1.92.3 [295af30f] Revise v3.1.17 [3646fa90] ScikitLearn v0.6.4 [ade2ca70] Dates

[ablaom]

@vishalhedgevantage Thanks for isolating this bug - much appreciated; I can reproduce.

I'm not sure of a resolution to this yet, but I have found the following workaround:

function test()
    T = (MLJ.@load RandomForestClassifier pkg=DecisionTree verbosity=0)
    Base.invokelatest(T)
end

cf. https://discourse.julialang.org/t/running-in-world-age-x-while-current-world-is-y-errors/5871

[ablaom]

@DilumAluthge I wonder if you have any suggestions here?

[DilumAluthge]

Hmmm. I'm not 100% sure. You can use invokelatest as a workaround, but it would be ideal if we could figure out why this is happening in the first place.

[ablaom]

After looking into this further, trying to load code from function scope is a tricky problem, and I'm going to propose we consider it beyond the reasonable remit of @load. Most everywhere in Julia, the expectation is that code is loaded directly from the top-level global scope, and @load will work from the top-level scope of any module, including a package module, which I believe suffices.

Happy to re-open if someone can make a good use-case for addressing and/or has some clue on how to make this work.