search

abhishekkrthakur / autoxgb

XGBoost + Optuna
Apache License 2.0
653 stars 90 forks source link

AttributeError: dlsym(0x7fd108ca6760, XGDMatrixCreateFromDense): symbol not found #4

Closed chandrad closed 2 years ago

chandrad commented 2 years ago

Hi

As per the subject, I am getting the error when I am running in local:


2021-11-01 15:45:04.651 | INFO     | autoxgb.autoxgb:__post_init__:42 - Output directory: output3
2021-11-01 15:45:04.652 | WARNING  | autoxgb.autoxgb:__post_init__:49 - No id column specified. Will default to `id`.
2021-11-01 15:45:04.653 | INFO     | autoxgb.autoxgb:_process_data:149 - Reading training data
2021-11-01 15:45:04.885 | INFO     | autoxgb.utils:reduce_memory_usage:48 - Mem. usage decreased to 2.19 Mb (76.0% reduction)
2021-11-01 15:45:04.891 | INFO     | autoxgb.autoxgb:_determine_problem_type:140 - Problem type: multi_class_classification
2021-11-01 15:45:04.892 | INFO     | autoxgb.autoxgb:_create_folds:58 - Creating folds
2021-11-01 15:45:04.922 | INFO     | autoxgb.autoxgb:_process_data:170 - Encoding target(s)
2021-11-01 15:45:04.931 | INFO     | autoxgb.autoxgb:_process_data:195 - Found 0 categorical features.
2021-11-01 15:45:05.054 | INFO     | autoxgb.autoxgb:_process_data:236 - Model config: train_filename='train.csv' test_filename=None idx='id' targets=['label'] problem_type=<ProblemType.multi_class_classification: 2> output='output3' features=['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'y1', 'z1', 'z2', 'z3', 'z4'] num_folds=5 use_gpu=False seed=42 categorical_features=[] num_trials=100 time_limit=360 fast=False
2021-11-01 15:45:05.054 | INFO     | autoxgb.autoxgb:_process_data:237 - Saving model config
2021-11-01 15:45:05.055 | INFO     | autoxgb.autoxgb:_process_data:241 - Saving encoders
[I 2021-11-01 15:45:05,230] A new study created in RDB with name: autoxgb
[W 2021-11-01 15:45:05,339] Trial 0 failed because of the following error: AttributeError('dlsym(0x7fd108ca6760, XGDMatrixCreateFromDense): symbol not found')
Traceback (most recent call last):
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/autoxgb/utils.py", line 172, in optimize
    model.fit(
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py", line 506, in inner_f
    return f(**kwargs)
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py", line 1231, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py", line 286, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py", line 1245, in <lambda>
    create_dmatrix=lambda **kwargs: DMatrix(nthread=self.n_jobs, **kwargs),
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py", line 506, in inner_f
    return f(**kwargs)
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py", line 616, in __init__
    handle, feature_names, feature_types = dispatch_data_backend(
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py", line 707, in dispatch_data_backend
    return _from_pandas_df(data, enable_categorical, missing, threads,
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py", line 299, in _from_pandas_df
    return _from_numpy_array(data, missing, nthread, feature_names,
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py", line 179, in _from_numpy_array
    _LIB.XGDMatrixCreateFromDense(
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/ctypes/__init__.py", line 386, in __getattr__
    func = self.__getitem__(name)
  File "/Users/A124661/opt/anaconda3/envs/deep_py38/lib/python3.8/ctypes/__init__.py", line 391, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fd108ca6760, XGDMatrixCreateFromDense): symbol not found
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/pp/ym01m3sx0hg3my_gzpsdl8680000gp/T/ipykernel_728/1462055845.py in <module>
     16     fast=fast,
     17 )
---> 18 axgb.train()

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/autoxgb/autoxgb.py in train(self)
    245     def train(self):
    246         self._process_data()
--> 247         best_params = train_model(self.model_config)
    248         logger.info("Training complete")
    249         self.predict(best_params)

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/autoxgb/utils.py in train_model(model_config)
    211         load_if_exists=True,
    212     )
--> 213     study.optimize(optimize_func, n_trials=model_config.num_trials, timeout=model_config.time_limit)
    214     return study.best_params
    215 

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/study.py in optimize(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
    398             )
    399 
--> 400         _optimize(
    401             study=self,
    402             func=func,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/_optimize.py in _optimize(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
     64     try:
     65         if n_jobs == 1:
---> 66             _optimize_sequential(
     67                 study,
     68                 func,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/_optimize.py in _optimize_sequential(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)
    161 
    162         try:
--> 163             trial = _run_trial(study, func, catch)
    164         except Exception:
    165             raise

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/_optimize.py in _run_trial(study, func, catch)
    262 
    263     if state == TrialState.FAIL and func_err is not None and not isinstance(func_err, catch):
--> 264         raise func_err
    265     return trial
    266 

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/optuna/study/_optimize.py in _run_trial(study, func, catch)
    211 
    212     try:
--> 213         value_or_values = func(trial)
    214     except exceptions.TrialPruned as e:
    215         # TODO(mamu): Handle multi-objective cases.

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/autoxgb/utils.py in optimize(trial, xgb_model, use_predict_proba, eval_metric, model_config)
    170 
    171         else:
--> 172             model.fit(
    173                 xtrain,
    174                 ytrain,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py in inner_f(*args, **kwargs)
    504         for k, arg in zip(sig.parameters, args):
    505             kwargs[k] = arg
--> 506         return f(**kwargs)
    507 
    508     return inner_f

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py in fit(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights, callbacks)
   1229 
   1230         model, feval, params = self._configure_fit(xgb_model, eval_metric, params)
-> 1231         train_dmatrix, evals = _wrap_evaluation_matrices(
   1232             missing=self.missing,
   1233             X=X,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py in _wrap_evaluation_matrices(missing, X, y, group, qid, sample_weight, base_margin, feature_weights, eval_set, sample_weight_eval_set, base_margin_eval_set, eval_group, eval_qid, create_dmatrix, enable_categorical, label_transform)
    284 
    285     """
--> 286     train_dmatrix = create_dmatrix(
    287         data=X,
    288         label=label_transform(y),

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/sklearn.py in <lambda>(**kwargs)
   1243             eval_group=None,
   1244             eval_qid=None,
-> 1245             create_dmatrix=lambda **kwargs: DMatrix(nthread=self.n_jobs, **kwargs),
   1246             enable_categorical=self.enable_categorical,
   1247             label_transform=label_transform,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py in inner_f(*args, **kwargs)
    504         for k, arg in zip(sig.parameters, args):
    505             kwargs[k] = arg
--> 506         return f(**kwargs)
    507 
    508     return inner_f

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/core.py in __init__(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, group, qid, label_lower_bound, label_upper_bound, feature_weights, enable_categorical)
    614             return
    615 
--> 616         handle, feature_names, feature_types = dispatch_data_backend(
    617             data,
    618             missing=self.missing,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py in dispatch_data_backend(data, missing, threads, feature_names, feature_types, enable_categorical)
    705         return _from_tuple(data, missing, threads, feature_names, feature_types)
    706     if _is_pandas_df(data):
--> 707         return _from_pandas_df(data, enable_categorical, missing, threads,
    708                                feature_names, feature_types)
    709     if _is_pandas_series(data):

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py in _from_pandas_df(data, enable_categorical, missing, nthread, feature_names, feature_types)
    297     data, feature_names, feature_types = _transform_pandas_df(
    298         data, enable_categorical, feature_names, feature_types)
--> 299     return _from_numpy_array(data, missing, nthread, feature_names,
    300                              feature_types)
    301 

~/opt/anaconda3/envs/deep_py38/lib/python3.8/site-packages/xgboost/data.py in _from_numpy_array(data, missing, nthread, feature_names, feature_types)
    177     config = bytes(json.dumps(args), "utf-8")
    178     _check_call(
--> 179         _LIB.XGDMatrixCreateFromDense(
    180             _array_interface(data),
    181             config,

~/opt/anaconda3/envs/deep_py38/lib/python3.8/ctypes/__init__.py in __getattr__(self, name)
    384         if name.startswith('__') and name.endswith('__'):
    385             raise AttributeError(name)
--> 386         func = self.__getitem__(name)
    387         setattr(self, name, func)
    388         return func

~/opt/anaconda3/envs/deep_py38/lib/python3.8/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    389 
    390     def __getitem__(self, name_or_ordinal):
--> 391         func = self._FuncPtr((name_or_ordinal, self))
    392         if not isinstance(name_or_ordinal, int):
    393             func.__name__ = name_or_ordinal

AttributeError: dlsym(0x7fd108ca6760, XGDMatrixCreateFromDense): symbol not found
abhishekkrthakur commented 2 years ago

Could you please post platform and xgboost version? are you able to use xgboost generally?

chandrad commented 2 years ago

You are right @abhishekkrthakur I reinstalled xgboost. Now it is working. I am closing this post.