【SOLVED-QUESTION】RuntimeError: [address=0.0.0.0:36297, pid=1139] Failed to import transformers.generation.utils

[sisuad]

问题描述： 可以通过localhost:9997加载出web ui，但是在注册本地模型或者加载内置模型（包括LLM和embedding），下载完成后运行时都会报这个错 RuntimeError: [address=0.0.0.0:36297, pid=1139] Failed to import transformers.generation.utils because of the following error (look up to see its traceback): /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

另外xinference-local这个命令提示不存在。

[aresnow1]

完整的报错发一下，并且发一下 pip show xinference, pip show torch, pip show transformers 这三个库的版本信息。

[sisuad]

pip show xinference Name: xinference Version: 0.7.3.1 Summary: Model Serving Made Easy Home-page: https://github.com/xorbitsai/inference Author: Qin Xuye Author-email: qinxuye@xprobe.io License: Apache License 2.0 Location: /usr/local/lib/python3.8/dist-packages Requires: modelscope, pydantic, typing-extensions, requests, fastapi, uvicorn, fsspec, openai, xoscar, torch, gradio, sse-starlette, tabulate, s3fs, tqdm, huggingface-hub, click Required-by:

pip show torch Name: torch Version: 2.1.0 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email: packages@pytorch.org License: BSD-3 Location: /usr/local/lib/python3.8/dist-packages Requires: nvidia-cuda-cupti-cu12, triton, fsspec, nvidia-cufft-cu12, nvidia-nccl-cu12, nvidia-cublas-cu12, networkx, nvidia-cuda-runtime-cu12, typing-extensions, nvidia-cusparse-cu12, filelock, nvidia-curand-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cusolver-cu12, nvidia-nvtx-cu12, nvidia-cudnn-cu12, sympy, jinja2 Required-by: xinference, xformers, vllm, torchvision, torchtext, torch-tensorrt, timm, sentence-transformers, controlnet-aux, accelerate

pip show transformers Name: transformers Version: 4.35.0 Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow Home-page: https://github.com/huggingface/transformers Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors) Author-email: transformers@huggingface.co License: Apache 2.0 License Location: /usr/local/lib/python3.8/dist-packages Requires: packaging, numpy, pyyaml, tqdm, safetensors, huggingface-hub, regex, requests, tokenizers, filelock Required-by: vllm, transformers-stream-generator, sentence-transformers

上面截图是0.6.x的版本，今早升级到0.7.3.1最新版，也是这个报错。麻烦了

完整报错

`2023-12-28 02:55:35,699 xinference.model.llm.llm_family 1740 INFO Caching from Modelscope: ZhipuAI/chatglm3-6b-32k 2023-12-28 02:55:35,700 xinference.model.llm.llm_family 1740 INFO Cache /root/.xinference/cache/chatglm3-32k-pytorch-6b exists 2023-12-28 02:55:38,542 xinference.core.worker 1740 ERROR Failed to load model 1-1-0 Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1345, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 84, in from accelerate.hooks import AlignDevicesHook, add_hook_to_module File "/usr/local/lib/python3.8/dist-packages/accelerate/init.py", line 3, in from .accelerator import Accelerator File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 36, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "/usr/local/lib/python3.8/dist-packages/accelerate/checkpointing.py", line 24, in from .utils import ( File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/init.py", line 139, in from .launch import ( File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/launch.py", line 32, in from ..utils.other import is_port_in_use, merge_dicts File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/other.py", line 32, in from .transformer_engine import convert_model File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/transformer_engine.py", line 21, in import transformer_engine.pytorch as te File "/usr/local/lib/python3.8/dist-packages/transformer_engine/init.py", line 7, in from . import pytorch File "/usr/local/lib/python3.8/dist-packages/transformer_engine/pytorch/init.py", line 6, in from .module import LayerNormLinear File "/usr/local/lib/python3.8/dist-packages/transformer_engine/pytorch/module.py", line 16, in import transformer_engine_extensions as tex ImportError: /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/xinference/core/worker.py", line 375, in launch_builtin_model await model_ref.load() File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 657, in send result = await self._run_coro(message.message_id, coro) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 368, in _run_coro return await coro File "/usr/local/lib/python3.8/dist-packages/xoscar/api.py", line 306, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive__ raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 524, in xoscar.core._BaseActor.__on_receive result = func(*args, kwargs) File "/usr/local/lib/python3.8/dist-packages/xinference/core/model.py", line 176, in load self._model.load() File "/usr/local/lib/python3.8/dist-packages/xinference/model/llm/pytorch/core.py", line 182, in load self._model, self._tokenizer = self._load_model(kwargs) File "/usr/local/lib/python3.8/dist-packages/xinference/model/llm/pytorch/chatglm.py", line 63, in _load_model model = AutoModel.from_pretrained( File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 553, in from_pretrained model_class = get_class_from_dynamic_module( File "/usr/local/lib/python3.8/dist-packages/transformers/dynamic_module_utils.py", line 499, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module.replace(".py", "")) File "/usr/local/lib/python3.8/dist-packages/transformers/dynamic_module_utils.py", line 199, in get_class_in_module module = importlib.import_module(module_path) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-32k-pytorch-6b/modeling_chatglm.py", line 24, in from transformers.modeling_utils import PreTrainedModel File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 41, in from .generation import GenerationConfig, GenerationMixin File "", line 1039, in _handle_fromlist File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1335, in getattr module = self._get_module(self._class_to_module[name]) File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1347, in _get_module raise RuntimeError( RuntimeError: [address=0.0.0.0:40217, pid=2216] Failed to import transformers.generation.utils because of the following error (look up to see its traceback): /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE 2023-12-28 02:55:38,578 xinference.api.restful_api 1705 ERROR [address=0.0.0.0:40217, pid=2216] Failed to import transformers.generation.utils because of the following error (look up to see its traceback): /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1345, in _get_module return importlib.import_module("." + module_name, self.name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 84, in from accelerate.hooks import AlignDevicesHook, add_hook_to_module File "/usr/local/lib/python3.8/dist-packages/accelerate/init.py", line 3, in from .accelerator import Accelerator File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 36, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "/usr/local/lib/python3.8/dist-packages/accelerate/checkpointing.py", line 24, in from .utils import ( File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/init.py", line 139, in from .launch import ( File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/launch.py", line 32, in from ..utils.other import is_port_in_use, merge_dicts File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/other.py", line 32, in from .transformer_engine import convert_model File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/transformer_engine.py", line 21, in import transformer_engine.pytorch as te File "/usr/local/lib/python3.8/dist-packages/transformer_engine/init.py", line 7, in from . import pytorch File "/usr/local/lib/python3.8/dist-packages/transformer_engine/pytorch/init__.py", line 6, in from .module import LayerNormLinear File "/usr/local/lib/python3.8/dist-packages/transformer_engine/pytorch/module.py", line 16, in import transformer_engine_extensions as tex ImportError: /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/xinference/api/restful_api.py", line 429, in launch_model model_uid = await (await self._get_supervisor_ref()).launch_builtin_model( File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 657, in send result = await self._run_coro(message.message_id, coro) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 368, in _run_coro return await coro File "/usr/local/lib/python3.8/dist-packages/xoscar/api.py", line 306, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive__ raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive result = await result File "/usr/local/lib/python3.8/dist-packages/xinference/core/supervisor.py", line 492, in launch_builtin_model await _launch_one_model(rep_model_uid) File "/usr/local/lib/python3.8/dist-packages/xinference/core/supervisor.py", line 461, in _launch_one_model await worker_ref.launch_builtin_model( File "xoscar/core.pyx", line 284, in pyx_actor_method_wrapper async with lock: File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper result = await result File "/usr/local/lib/python3.8/dist-packages/xinference/core/utils.py", line 33, in wrapped ret = await func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/xinference/core/worker.py", line 375, in launch_builtin_model await model_ref.load() File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 657, in send result = await self._run_coro(message.message_id, coro) File "/usr/local/lib/python3.8/dist-packages/xoscar/backends/pool.py", line 368, in _run_coro return await coro File "/usr/local/lib/python3.8/dist-packages/xoscar/api.py", line 306, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.__on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive__ with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 524, in xoscar.core._BaseActor.on_receive result = func(*args, kwargs) File "/usr/local/lib/python3.8/dist-packages/xinference/core/model.py", line 176, in load self._model.load() File "/usr/local/lib/python3.8/dist-packages/xinference/model/llm/pytorch/core.py", line 182, in load self._model, self._tokenizer = self._load_model(kwargs) File "/usr/local/lib/python3.8/dist-packages/xinference/model/llm/pytorch/chatglm.py", line 63, in _load_model model = AutoModel.from_pretrained( File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 553, in from_pretrained model_class = get_class_from_dynamic_module( File "/usr/local/lib/python3.8/dist-packages/transformers/dynamic_module_utils.py", line 499, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module.replace(".py", "")) File "/usr/local/lib/python3.8/dist-packages/transformers/dynamic_module_utils.py", line 199, in get_class_in_module module = importlib.import_module(module_path) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 848, in exec_module File "", line 219, in _call_with_frames_removed File "/root/.cache/huggingface/modules/transformers_modules/chatglm3-32k-pytorch-6b/modeling_chatglm.py", line 24, in from transformers.modeling_utils import PreTrainedModel File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 41, in from .generation import GenerationConfig, GenerationMixin File "", line 1039, in _handle_fromlist File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1335, in getattr module = self._get_module(self._class_to_module[name]) File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1347, in _get_module raise RuntimeError( RuntimeError: [address=0.0.0.0:40217, pid=2216] Failed to import transformers.generation.utils because of the following error (look up to see its traceback): /usr/local/lib/python3.8/dist-packages/transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEE `

[trexliu]

+1 有啥解法么

[aresnow1]

@trexliu @sisuad 可以确认下 pydantic 的版本吗，看上去是 transformers 和 pydantic>2 的版本有兼容性问题，pip install "pydantic<2" 安装小于 2 的版本试一下。

[sisuad]

pip show pydantic Name: pydantic Version: 1.10.0 Summary: Data validation and settings management using python type hints Home-page: https://github.com/pydantic/pydantic Author: Samuel Colvin Author-email: s@muelcolvin.com License: MIT Location: /usr/local/lib/python3.8/dist-packages Requires: typing-extensions Required-by: xinference, vllm, thinc, spacy, openai, gradio, fastapi, confection 确认了，还是一样的问题

[ChengjieLi28]

@sisuad /usr/local/lib/python3.8根据你的日志，你使用的python环境就是系统的3.8环境吗？

[ChengjieLi28]

@sisuad /usr/local/lib/python3.8根据你的日志，你使用的python环境就是系统的3.8环境吗？

系统默认的python环境很可能缺少什么so之类的，我推荐使用anaconda或者miniconda创建python环境，再试试

[codingl2k1]

Similar issue: https://github.com/moymix/TaskMatrix/issues/116

[sisuad]

Similar issue: moymix/TaskMatrix#116

thanks a lots, that's actually my problem. I used the solution to solve this issue

bypip uninstall transformer-engine

by the way, I use the third-party docker images from apecloud/xinference:latest-amd64

[github-actions[bot]]

This issue is stale because it has been open for 7 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 5 days since being marked as stale.