ModuleNotFoundError: No module named 'vllm.distributed'

[yangqing-yq]

Python 3.11.10 | packaged by conda-forge | (main, Sep 22 2024, 14:10:38) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from vllm import SamplingParams from ipex_llm.vllm.xpu.engine import IPEXLLMClass as LLM /home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpeg or libpng installed before building torchvision from source? warn( 2024-09-30 18:40:24,346 - INFO - intel_extension_for_pytorch auto imported Traceback (most recent call last): File "", line 1, in File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/engine/init.py", line 16, in from .engine import IPEXLLMAsyncLLMEngine, IPEXLLMLLMEngine, IPEXLLMClass File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/engine/engine.py", line 22, in from ipex_llm.vllm.xpu.model_convert import _ipex_llm_convert File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/model_convert.py", line 18, in from vllm.distributed import tensor_model_parallel_gather, tensor_model_parallel_all_gather File "/home/wxf/miniforge3/envs/llm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'vllm.distributed'

[QuentinVitt]

+1

python 3.11.10 | followed the instructions for Install ipex-llm and and then followed the Install vllm instructions. I used the following fix for a problem I had: https://github.com/pytorch/pytorch/issues/123097#issuecomment-2050934427 and now have the same error:

Traceback (most recent call last): File "/home/quentin-vitt/vllm/offline_inference.py", line 35, in <module> from ipex_llm.vllm.xpu.engine import IPEXLLMClass as LLM File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/engine/__init__.py", line 16, in <module> from .engine import IPEXLLMAsyncLLMEngine, IPEXLLMLLMEngine, IPEXLLMClass File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/engine/engine.py", line 22, in <module> from ipex_llm.vllm.xpu.model_convert import _ipex_llm_convert File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/vllm/xpu/model_convert.py", line 18, in <module> from vllm.distributed import tensor_model_parallel_gather, tensor_model_parallel_all_gather File "/home/quentin-vitt/miniforge3/envs/ipex-vllm/lib/python3.11/site-packages/ipex_llm/utils/ipex_importer.py", line 76, in custom_ipex_import return RAW_IMPORT(name, globals, locals, fromlist, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'vllm.distributed'

[xiangyuT]

Hi @yangqing-yq @QuentinVitt, There is a recent update for out vllm repo but the vllm installation instruction in our quickstart is not updated. The current branch for vllm is 0.5.4 instead of sycl_xpu. To update this, you could try using our docker image intelanalytics/ipex-llm-serving-xpu:2.2.0-SNAPSHOT or reinstall by these commands:

conda activate ipex-vllm
source /opt/intel/oneapi/setvars.sh
git clone -b 0.5.4 https://github.com/analytics-zoo/vllm.git
cd vllm
pip install -r requirements-xpu.txt
VLLM_TARGET_DEVICE=xpu python setup.py install
pip install mpi4py fastapi uvicorn openai