[BUG] Random Forest with Pycaret on gpu throws exception : Could not load the correct number of nodes

Describe the bug Random Forest with Pycaret on gpu throws exception : Could not load the correct number of nodes

Steps/Code to reproduce bug Install Pycaret rapidsai and cuml using conda

Code

import pandas as pd
from imblearn.over_sampling import ADASYN
from pycaret.classification import automl, compare_models, finalize_model, pull, setup, create_model
import pycaret_extensions as pycaret_ext

def main():
    df = pd.read_csv("data/hour.csv", sep=",").drop_duplicates(subset=None).dropna()
    df["cnt"] = pd.cut(df["cnt"], bins=[0, 500, 999], labels=[0, 1]).astype("int")
    setup(
        data=df,
        target=-1,  
        normalize=True,
        normalize_method="robust",
        transformation=True,
        transformation_method="quantile",
        fix_imbalance=False,
        fix_imbalance_method=ADASYN(sampling_strategy="minority"),  
        remove_multicollinearity=True,  
        train_size=0.75,  
        fold=3, 
        log_experiment=False,  
        session_id=1,  
        system_log=True,
        use_gpu=True,
    )
    names = ["rf"]

    top5: list[object] = compare_models(n_select=5, sort="Accuracy",include=names)

Result

Processing: 0%| | 0/9 [00:00<?, ?it/s] Then the program stops. I was able to get the exception text while debugging only.

The dataset used (hour.csv) is freely available at https://rubikscode.net/2021/07/19/top-23-best-public-datasets-for-practicing-machine-learning/ under Bike Sharing Demand Dataset. I tried with Wine quality dataset too and got same bug.

Expected behavior Processing continue as expected. Note : The code works if you remove the use_gpu flag in setup().

Environment details (please complete the following information):

See file for env details env.txt

Additional context Stack Trace : File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 76, in _cached_call return cache[method] KeyError: 'predict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 794, in compare_models model, model_fit_time = self._create_model(create_model_args) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 1533, in _create_model model, model_fit_time, modelresults, = self._create_model_with_cv( File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 1126, in _create_model_with_cv scores = cross_validate( File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 578, in safe_patch_function patch_function(call_original, *args, *kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/sklearn/init.py", line 1875, in patched_fn_with_autolog_disabled return original(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 559, in call_original return call_original_fn_with_event_logging(_original_fn, og_args, og_kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 494, in call_original_fn_with_event_logging original_fn_result = original_fn(*og_args, og_kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 556, in _original_fn original_result = original(*_og_args, *_og_kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 266, in cross_validate results = parallel( File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/utils/parallel.py", line 63, in call return super().call(iterable_with_config) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 1085, in call if self.dispatch_one_batch(iterator): File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch self._dispatch(tasks) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 819, in _dispatch job = self._backend.apply_async(batch, callback=cb) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 208, in apply_async result = ImmediateResult(func) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 597, in init self.results = batch() File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 288, in call return [func(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 288, in return [func(*args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/utils/parallel.py", line 123, in call return self.function(*args, *kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 124, in fit_and_score return wrapper(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 122, in wrapper return _fit_and_score(*args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 708, in _fit_and_score test_scores = _score(estimator, X_test, y_test, scorer, error_score) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 144, in wrapper return f(args[0], *tuple(args[1:]), *kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 767, in _score scores = scorer(estimator, X_test, y_test) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 121, in call raise e File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 115, in call score = scorer._score(cached_call, estimator, args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 276, in _score y_pred = method_caller(estimator, "predict", X) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 78, in _cached_call result = getattr(estimator, method)(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pipeline.py", line 327, in predict y = self.steps[-1][-1].predict(X, predict_params) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper return func(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/cuml_wrappers.py", line 130, in predict X = super().predict(X, *args, *kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper ret = func(args, kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/nvtx/nvtx.py", line 115, in inner result = func(*args, *kwargs) File "randomforestclassifier.pyx", line 605, in cuml.ensemble.randomforestclassifier.RandomForestClassifier.predict File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper ret = func(args, **kwargs) File "randomforest_common.pyx", line 348, in cuml.ensemble.randomforest_common.BaseRandomForestModel._predict_model_on_gpu File "randomforest_common.pyx", line 216, in cuml.ensemble.randomforest_common.BaseRandomForestModel._obtain_treelite_handle File "randomforest_shared.pyx", line 120, in cuml.ensemble.randomforest_shared.treelite_deserialize File "randomforest_shared.pyx", line 104, in cuml.ensemble.randomforest_shared.init_from_frames File "randomforest_shared.pyx", line 83, in cuml.ensemble.randomforest_shared._init_from_frames RuntimeError: Could not load the correct number of nodes

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/datadrive/src/train_automl.py", line 92, in main() File "/datadrive/src/train_automl.py", line 68, in main top5: list[object] = compare_models(n_select=5, sort="Accuracy", errors="raise") File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/utils/generic.py", line 965, in wrapper return func(*args, **kwargs) File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/classification/functional.py", line 814, in compare_models return _CURRENT_EXPERIMENT.compare_models( File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/classification/oop.py", line 1180, in compare_models return_values = super().compare_models( File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 806, in compare_models raise RuntimeError( RuntimeError: create_model() failed for model rf. RuntimeError: Could not load the correct number of nodes

rapidsai / cuml

[BUG] Random Forest with Pycaret on gpu throws exception : Could not load the correct number of nodes #5801