beyondguo / LLM-Tuning

Tuning LLMs with no tears💦; Sample Design Engineering (SDE) for more efficient downstream-tuning.
956 stars 98 forks source link

ImportError: dlopen: cannot load any more object with static TLS #59

Open hjing100 opened 4 months ago

hjing100 commented 4 months ago

Traceback (most recent call last): File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/__check_build/init.py", line 45, in from ._check_build import check_build # noqa ImportError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1184, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/local/anaconda3/envs/test39/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 790, in exec_module File "", line 228, in _call_with_frames_removed File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/pipelines/init.py", line 62, in from .document_question_answering import DocumentQuestionAnsweringPipeline File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/pipelines/document_question_answering.py", line 29, in from .question_answering import select_starts_ends File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/pipelines/question_answering.py", line 9, in from ..data import SquadExample, SquadFeatures, squad_convert_examples_to_features File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/data/init.py", line 26, in from .metrics import glue_compute_metrics, xnli_compute_metrics File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/data/metrics/init.py", line 20, in from sklearn.metrics import f1_score, matthews_corrcoef File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/init.py", line 83, in from . import ( File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/check_build/init.py", line 47, in raise_build_error(e) File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/__check_build/init__.py", line 31, in raise_build_error raise ImportError("""%s ImportError: dlopen: cannot load any more object with static TLS


Contents of /usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/check_build: init.py _check_build.cpython-39-x86_64-linux-gnu.sopycache__


It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget to build the package before using it: run python setup.py install or make in the source directory.

If you have used an installer, please check that it is suited for your Python version, your operating system and your platform.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/data/zhj/projects/LLM-Tuning-master/RLHF/reward_modeling.py", line 11, in import evaluate File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/evaluate/init.py", line 29, in from .evaluation_suite import EvaluationSuite File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/evaluate/evaluation_suite/init.py", line 10, in from ..evaluator import evaluator File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/evaluate/evaluator/init.py", line 27, in from .audio_classification import AudioClassificationEvaluator File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/evaluate/evaluator/audio_classification.py", line 23, in from .base import EVALUATOR_COMPUTE_RETURN_DOCSTRING, EVALUTOR_COMPUTE_START_DOCSTRING, Evaluator File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/evaluate/evaluator/base.py", line 34, in from transformers import Pipeline, pipeline File "", line 1055, in _handle_fromlist File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1174, in getattr module = self._get_module(self._class_to_module[name]) File "/usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1186, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback): dlopen: cannot load any more object with static TLS


Contents of /usr/local/anaconda3/envs/test39/lib/python3.9/site-packages/sklearn/check_build: init.py _check_build.cpython-39-x86_64-linux-gnu.sopycache__


It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget to build the package before using it: run python setup.py install or make in the source directory.

If you have used an installer, please check that it is suited for your Python version, your operating system and your platform.

说明: 最开始是:accuracy = evaluate.load("accuracy")报错 按照了accuracy,还是报错,就自己下载了https://github.com/huggingface/datasets/tree/main/metrics下载了accuracy 但是报错没有sklearn pip install scikit-learn 然后就报以上错误,请问有人遇到过吗?

hjing100 commented 4 months ago

解决方案:升级glibc版本到2.23。