search

ray-project / tune-sklearn

A drop-in replacement for Scikit-Learn’s GridSearchCV / RandomizedSearchCV -- but with cutting edge hyperparameter tuning techniques.
https://docs.ray.io/en/master/tune/api_docs/sklearn.html
Apache License 2.0
465 stars 52 forks source link

AttributeError: 'TuneGridSearchCV' object has no attribute 'best_estimator_' #146

Closed andyolivers closed 3 years ago

andyolivers commented 3 years ago

Hi! I'm getting the following error when running a TuneGridSearchCV:

AttributeError: 'TuneGridSearchCV' object has no attribute 'bestestimator'

My settings are the following: `
model = GradientBoostingRegressor() pipe = Pipeline([('model', model)])

Hyperparameter tuning

    gsearch = TuneGridSearchCV(pipe, cv=my_cv, param_grid=param_search, n_jobs=7, refit=True,
                           scoring='neg_mean_absolute_error', verbose=True, early_stopping=False)`

Could you help? Many thanks in advance.

richardliaw commented 3 years ago

hmm does this error appear on master?

andyolivers commented 3 years ago

yes

richardliaw commented 3 years ago

@andyolivers can you provide a simple repro script that I can run?

cc @Yard1 can you take a look?

Yard1 commented 3 years ago

Yeah, a reproducible example would be very helpful here. I think that something is breaking during optimization. The lack of best_estimator_ usually means that there has been an exception on the way somewhere.

andyolivers commented 3 years ago

This is company code, so I'm really can't get much into details. Below is my build_model function and the model parameters:

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.model_selection import TimeSeriesSplit
from sklearn.model_selection import GridSearchCV
#from ray.tune.sklearn import TuneGridSearchCV
from tune_sklearn import TuneGridSearchCV
from lib.ModelLib import xgboost
from lib.ModelLib import ridge
from lib.ModelLib import lasso
from lib.ModelLib import randomforestreg
from lib.ModelLib import elasticnet
from lib.ModelLib import gradientboosting
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from lib.feature_engineering import create_lagged_variables
from lib.feature_engineering import add_rolling_variable
import warnings
warnings.filterwarnings('ignore')

`
def build_model(X, y, cv,n_splits, model_list=list):
    #model_list = [xgboost,ridge,lasso,randomforestreg,elasticnet]

gsearchs = []
scores = []
best_params = []
cv_results = []
best_models = []
for i in model_list:
    print(i)

    # Get model
    model = i.model

    # Create a pipeline
    # pipe = Pipeline([('scaler', MinMaxScaler()), ('model', xgb.XGBRegressor())])
    pipe = Pipeline([('model', model)])

    # Get hyperparameter grid
    param_search = i.parameters

    if str(i) == '''<class 'lib.ModelLib.xgboost'>''':
        pass
    else:
        X = X.fillna(0)

    # Cross-validation
    tscv = TimeSeriesSplit(n_splits=n_splits)
    my_cv = [(train, test) for train, test in tscv.split(X)]
    # my_cv = cv

    # Hyperparameter tuning

    gsearch = TuneGridSearchCV(pipe, cv=my_cv, param_grid=param_search, n_jobs=7, refit=True,
                           scoring='neg_mean_absolute_error', verbose=True, early_stopping=False)

    '''gsearch = GridSearchCV(pipe, cv=my_cv, param_grid=param_search, n_jobs=7,
                           scoring='neg_mean_absolute_error', verbose=True)'''
    gsearch.fit(X, y)

    gsearchs.append(gsearch)
    best_models.append(gsearch.best_estimator_)
    scores.append(abs(gsearch.best_score_))
    best_params.append(gsearch.best_params_)
    cv_results.append(gsearch.cv_results_)

    print('MAE for model {0}: {1}'.format(i, gsearch.best_score_))

return gsearchs, scores, best_params, cv_results, best_models

`

`class gradientboosting: model = GradientBoostingRegressor()

parameters = { 'model__loss': ['ls'],
                'model__learning_rate': [.01, 0.1, 0.5, 1],
                'model__max_depth': [4, 8, 12, 16, 20],
                'model__min_samples_leaf': [10, 30, 60, 90, 120],
                'model__min_samples_split': [30, 60, 90, 120],
                'model__subsample': [0.5, 0.8, 1],
                'model__n_estimators': [10, 25, 50, 100, 150],
                'model__random_state': [16],
                'model__verbose': [0]
               }

'''parameters = { 'model__loss': ['ls'],
                'model__learning_rate': [.01, 0.1, 0.5, 1],
                'model__max_depth': [4, 8, 12, 16, 20],
                'model__min_samples_leaf': [10, 30, 60, 90, 120],
                'model__min_samples_split': [30, 60, 90, 120],
                'model__subsample': [0.5, 0.8, 1],
                'model__n_estimators': [10, 25, 50, 100, 150],
                'model__random_state': [16],
                'model__max_features': [15, 30, 45, 60, 75],
                'model__verbose': [0]
               }'''

'''parameters = { 'model__loss': ['ls'],
                'model__learning_rate': [.01, 0.05, 1, 3],
                'model__max_depth': [3, 6, 9, 12],
                'model__min_samples_leaf': [3, 6, 9, 12],
                'model__min_samples_split': [3, 6, 9, 12],
                'model__subsample': [0.5, 1, 1.5],
                'model__n_estimators': [5, 10, 25, 50, 100, 150],
                'model__verbose': [0]
               }'''

@staticmethod
def best_model(params):
    clf = GradientBoostingRegressor(**params)
    return clf`
Yard1 commented 3 years ago

Are there any exceptions raised when optimizing?

andyolivers commented 3 years ago

Here:


Could not instantiate TBXLogger: No module named 'tensorboardX'.
== Status ==
Memory usage on this node: 6.3/8.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 2/8 CPUs, 0/0 GPUs, 0.0/2.2 GiB heap, 0.0/0.73 GiB objects
Result logdir: /Users/andre.oliveira/ray_results/_Trainable_2020-11-24_10-00-18
Number of trials: 1/6000 (1 RUNNING)
+------------------------+----------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+
| Trial name             | status   | loc   |   model__learning_rate | model__loss   |   model__max_depth |   model__min_samples_leaf |   model__min_samples_split |   model__n_estimators |   model__random_state |   model__subsample |   model__verbose |
|------------------------+----------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------|
| _Trainable_db54f_00000 | RUNNING  |       |                   0.01 | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
+------------------------+----------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+

(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) LokyProcess-4 failed with traceback: 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) Traceback (most recent call last):
(pid=55290)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55290)     prep_data = pickle.load(from_parent)
(pid=55290) ValueError: unsupported pickle protocol: 5
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-1 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) LokyProcess-2 failed with traceback: 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) Traceback (most recent call last):
(pid=55290)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55290)     prep_data = pickle.load(from_parent)
(pid=55290) ValueError: unsupported pickle protocol: 5
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-5 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) 
(pid=55290) LokyProcess-6 failed with traceback: 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) Traceback (most recent call last):
(pid=55290)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55290)     prep_data = pickle.load(from_parent)
(pid=55290) ValueError: unsupported pickle protocol: 5
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-5 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-4 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) LokyProcess-1 failed with traceback: 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) Traceback (most recent call last):
(pid=55288)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55288)     prep_data = pickle.load(from_parent)
(pid=55288) ValueError: unsupported pickle protocol: 5
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) LokyProcess-3 failed with traceback: 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) Traceback (most recent call last):
(pid=55288)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55288)     prep_data = pickle.load(from_parent)
(pid=55288) ValueError: unsupported pickle protocol: 5
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-6 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-3 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-7 failed with traceback: 
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-8 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) LokyProcess-2 failed with traceback: 
(pid=55286) --------------------------------------------------------------------------------
(pid=55286) Traceback (most recent call last):
(pid=55286)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55286)     prep_data = pickle.load(from_parent)
(pid=55286) ValueError: unsupported pickle protocol: 5
(pid=55286) 
(pid=55286) 
(pid=55286) --------------------------------------------------------------------------------
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) LokyProcess-8 failed with traceback: 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) Traceback (most recent call last):
(pid=55288)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55288)     prep_data = pickle.load(from_parent)
(pid=55288) ValueError: unsupported pickle protocol: 5
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55290) 
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-4 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) LokyProcess-1 failed with traceback: 
(pid=55290) --------------------------------------------------------------------------------
(pid=55290) Traceback (most recent call last):
(pid=55290)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55290)     prep_data = pickle.load(from_parent)
(pid=55290) ValueError: unsupported pickle protocol: 5
(pid=55290) 
(pid=55290) 
(pid=55290) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-1 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-2 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) LokyProcess-2 failed with traceback: 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) Traceback (most recent call last):
(pid=55288)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55288)     prep_data = pickle.load(from_parent)
(pid=55288) ValueError: unsupported pickle protocol: 5
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-7 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-3 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) LokyProcess-6 failed with traceback: 
(pid=55292) --------------------------------------------------------------------------------
(pid=55292) Traceback (most recent call last):
(pid=55292)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55292)     prep_data = pickle.load(from_parent)
(pid=55292) ValueError: unsupported pickle protocol: 5
(pid=55292) 
(pid=55292) 
(pid=55292) --------------------------------------------------------------------------------
Trial _Trainable_db54f_00001: Error processing event.
Traceback (most recent call last):
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 726, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 489, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/ray/worker.py", line 1452, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TerminatedWorkerError): ray::_Trainable.train() (pid=55290, ip=192.168.1.84)
  File "python/ray/_raylet.pyx", line 482, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 436, in ray._raylet.execute_task.function_executor
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/ray/tune/trainable.py", line 336, in train
    result = self.step()
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/tune_sklearn/_trainable.py", line 119, in step
    return self._train()
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/tune_sklearn/_trainable.py", line 237, in _train
    return_train_score=self.return_train_score,
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/sklearn/utils/validation.py", line 72, in inner_f
    return f(**kwargs)
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 248, in cross_validate
    for train, test in cv.split(X, y, groups))
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/parallel.py", line 1061, in __call__
    self.retrieve()
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/parallel.py", line 940, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {EXIT(1)}
== Status ==
Memory usage on this node: 5.8/8.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs, 0.0/2.2 GiB heap, 0.0/0.73 GiB objects
Result logdir: /Users/andre.oliveira/ray_results/_Trainable_2020-11-24_10-00-18
Number of trials: 5/6000 (1 ERROR, 4 TERMINATED)
+------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+
| Trial name             | status     | loc   |   model__learning_rate | model__loss   |   model__max_depth |   model__min_samples_leaf |   model__min_samples_split |   model__n_estimators |   model__random_state |   model__subsample |   model__verbose |
|------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------|
| _Trainable_db54f_00000 | TERMINATED |       |                   0.01 | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00002 | TERMINATED |       |                   0.5  | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00003 | TERMINATED |       |                   1    | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00004 | TERMINATED |       |                   0.01 | ls            |                  8 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00001 | ERROR      |       |                   0.1  | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
+------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+
Number of errored trials: 1
+------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                                                                                                                                                        |
|------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _Trainable_db54f_00001 |            1 | /Users/andre.oliveira/ray_results/_Trainable_2020-11-24_10-00-18/_Trainable_db54f_00001_1_model__learning_rate=0.1,model__loss=ls,model__max_depth=4,model__min_samples_leaf=10,model__min_samples__2020-11-24_10-00-18/error.txt |
+------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) LokyProcess-4 failed with traceback: 
(pid=55288) --------------------------------------------------------------------------------
(pid=55288) Traceback (most recent call last):
(pid=55288)   File "/Users/andre.oliveira/.conda/envs/main.py/lib/python3.7/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 197, in <module>
(pid=55288)     prep_data = pickle.load(from_parent)
(pid=55288) ValueError: unsupported pickle protocol: 5
(pid=55288) 
(pid=55288) 
(pid=55288) --------------------------------------------------------------------------------
== Status ==
Memory usage on this node: 5.8/8.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs, 0.0/2.2 GiB heap, 0.0/0.73 GiB objects
Result logdir: /Users/andre.oliveira/ray_results/_Trainable_2020-11-24_10-00-18
Number of trials: 5/6000 (1 ERROR, 4 TERMINATED)
+------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+
| Trial name             | status     | loc   |   model__learning_rate | model__loss   |   model__max_depth |   model__min_samples_leaf |   model__min_samples_split |   model__n_estimators |   model__random_state |   model__subsample |   model__verbose |
|------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------|
| _Trainable_db54f_00000 | TERMINATED |       |                   0.01 | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00002 | TERMINATED |       |                   0.5  | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00003 | TERMINATED |       |                   1    | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00004 | TERMINATED |       |                   0.01 | ls            |                  8 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
| _Trainable_db54f_00001 | ERROR      |       |                   0.1  | ls            |                  4 |                        10 |                         30 |                    10 |                    16 |                0.5 |                0 |
+------------------------+------------+-------+------------------------+---------------+--------------------+---------------------------+----------------------------+-----------------------+-----------------------+--------------------+------------------+
Number of errored trials: 1
+------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                                                                                                                                                        |
|------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _Trainable_db54f_00001 |            1 | /Users/andre.oliveira/ray_results/_Trainable_2020-11-24_10-00-18/_Trainable_db54f_00001_1_model__learning_rate=0.1,model__loss=ls,model__max_depth=4,model__min_samples_leaf=10,model__min_samples__2020-11-24_10-00-18/error.txt |
+------------------------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Traceback (most recent call last):
  File "/Users/andre.oliveira/Desktop/CS Ticket Forecast ML/main.py", line 155, in <module>
    gsearchs, scores, best_params, cv_results, best_models = build_model(X_train, y_train, cv, n_splits, model_list)
  File "/Users/andre.oliveira/Desktop/CS Ticket Forecast ML/lib/model_data.py", line 63, in build_model
    best_models.append(gsearch.best_estimator_)
AttributeError: 'TuneGridSearchCV' object has no attribute 'best_estimator_'

Process finished with exit code 1
Yard1 commented 3 years ago

Could you try adding sk_n_jobs=1 to TuneGridSearchCV arguments?

andyolivers commented 3 years ago

Now it is running fine. Is it really impossible to implement earlystopping with scikit's Gradient Boosting?

Yard1 commented 3 years ago

I believe early stopping for it is already supported?

andyolivers commented 3 years ago

I tried to implement with this model and got this error: ValueError: Early stopping is not supported because the estimator does not havepartial_fit, does not support warm_start, or is a tree classifier. Setearly_stopping=False. Then tried to remove the pipeline and use just the model and same error.

Yard1 commented 3 years ago

I'll take a look.

richardliaw commented 3 years ago

closing this, as the original issue is fixed!