search

intel-analytics / ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc.
Apache License 2.0
6.47k stars 1.24k forks source link

RuntimeError: PyTorch is not linked with support for xpu devices #11525

Open aoke79 opened 2 months ago

aoke79 commented 2 months ago

Dear, I created a new virtual env, and installed all the packages, however, when I run the code, it does not run on GPU (XPU). here is the log: ipex-llm\python\llm\dev\benchmark\all-in-one>python run.py C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\transformers\deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( 2024-07-08 12:55:55,928 - INFO - Note: NumExpr detected 22 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2024-07-08 12:55:55,928 - INFO - NumExpr defaulting to 8 threads. 2024-07-08 12:55:56,466 - INFO - Converting the current model to sym_int4 format...... Traceback (most recent call last): File "C:\AIGC\npu\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 1862, in run_model(model, api, in_out_pairs, conf['local_model_hub'], conf['warm_up'], conf['num_trials'], conf['num_beams'], File "C:\AIGC\npu\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 133, in run_model result = run_transformer_int4_gpu_win(repo_id, local_model_hub, in_out_pairs, warm_up, num_trials, num_beams, low_bit, cpu_embedding, batch_size, streaming) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AIGC\npu\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 912, in run_transformer_int4_gpu_win model = model.to('xpu') ^^^^^^^^^^^^^^^ File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\transformers\modeling_utils.py", line 2460, in to return super().to(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\torch\nn\modules\module.py", line 1160, in to return self._apply(convert) ^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\torch\nn\modules\module.py", line 833, in _apply param_applied = fn(param) ^^^^^^^^^ File "C:\ProgramData\anaconda3\envs\env_p311\Lib\site-packages\torch\nn\modules\module.py", line 1158, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: PyTorch is not linked with support for xpu devices

attached the pip freeze txt

aoke79 commented 2 months ago

pip.txt

qiuxin2012 commented 2 months ago

Have you installed Intel driver? See https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Overview/FAQ/faq.md#pytorch-is-not-linked-with-support-for-xpu-devices

qiuxin2012 commented 2 months ago

You can follow https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/scripts to verity your runtime environment.

Edward-Lin commented 2 months ago

please refer the "pip.txt", you will get what you need. :(

aoke79 commented 2 months ago

env-check

aoke79 commented 2 months ago

[Uploading check-log.txt…]()

aoke79 commented 2 months ago

please check the logs.

qiuxin2012 commented 2 months ago

I looks like you loading is not finished, I can't open it.

qiuxin2012 commented 2 months ago

env-check

It's a typo, I will fix it.

aoke79 commented 2 months ago

absl-py==2.1.0 accelerate==0.23.0 aiofiles==23.2.1 aiohttp==3.9.5 aiosignal==1.3.1 altair==5.3.0 annotated-types==0.6.0 antlr4-python3-runtime==4.9.3 anyio==4.3.0 attrs==23.2.0 bigdl-core-xe-21==2.5.0b20240707 bigdl-core-xe-addons-21==2.5.0b20240707 bigdl-core-xe-batch-21==2.5.0b20240707 bigdl-core-xe-esimd-21==2.5.0b20240520 certifi==2024.2.2 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 colorama==0.4.6 contourpy==1.2.1 cycler==0.12.1 dataclasses-json==0.6.4 DataProperty==1.0.1 datasets==2.19.0 dill==0.3.8 distro==1.9.0 dpcpp-cpp-rt==2024.0.2 faiss-cpu==1.8.0 fastapi==0.110.2 ffmpy==0.3.2 filelock==3.13.4 fonttools==4.51.0 frozenlist==1.4.1 fsspec==2024.3.1 gradio==4.28.3 gradio_client==0.16.0 greenlet==3.0.3 h11==0.14.0 httpcore==1.0.5 httpx==0.27.0 huggingface-hub==0.22.2 idna==3.7 importlib_resources==6.4.0 intel-cmplr-lib-rt==2024.0.2 intel-cmplr-lic-rt==2024.0.2 intel-extension-for-pytorch==2.1.10+xpu intel-opencl-rt==2024.0.2 intel-openmp==2024.0.2 ipex-llm==2.1.0b20240707 Jinja2==3.1.3 joblib==1.4.0 jsonlines==4.0.0 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 kiwisolver==1.4.5 langchain==0.1.16 langchain-community==0.0.34 langchain-core==0.1.46 langchain-text-splitters==0.0.1 langsmith==0.1.51 -e git+https://github.com/EleutherAI/lm-evaluation-harness.git@b281b0921b636bc36ad05c0b0b0763bd6dd43463#egg=lm_eval markdown-it-py==3.0.0 MarkupSafe==2.1.5 marshmallow==3.21.1 matplotlib==3.8.4 mbstrdecoder==1.1.3 mdurl==0.1.2 mkl==2024.0.0 mkl-dpcpp==2024.0.0 mpmath==1.3.0 multidict==6.0.5 multiprocess==0.70.16 mypy-extensions==1.0.0 networkx==3.3 nltk==3.8.1 numexpr==2.10.0 numpy==1.26.4 omegaconf==2.3.0 onednn==2024.0.0 onemkl-sycl-blas==2024.0.0 onemkl-sycl-datafitting==2024.0.0 onemkl-sycl-dft==2024.0.0 onemkl-sycl-lapack==2024.0.0 onemkl-sycl-rng==2024.0.0 onemkl-sycl-sparse==2024.0.0 onemkl-sycl-stats==2024.0.0 onemkl-sycl-vm==2024.0.0 openai==1.23.6 orjson==3.10.1 packaging==23.2 pandas==2.2.2 pathvalidate==3.2.0 peft==0.10.0 pillow==10.3.0 portalocker==2.8.2 protobuf==5.27.0rc1 psutil==5.9.8 py-cpuinfo==9.0.0 pyarrow==16.0.0 pyarrow-hotfix==0.6 pybind11==2.12.0 pycountry==23.12.11 pydantic==2.7.0 pydantic_core==2.18.1 pydub==0.25.1 Pygments==2.17.2 pyparsing==3.1.2 pypdf==4.2.0 pytablewriter==1.2.0 python-dateutil==2.9.0.post0 python-multipart==0.0.9 pytz==2024.1 pywin32==306 PyYAML==6.0.1 referencing==0.35.0 regex==2024.4.16 requests==2.31.0 rich==13.7.1 rouge-score==0.1.2 rpds-py==0.18.0 ruff==0.4.2 sacrebleu==1.5.0 safetensors==0.4.3 scikit-learn==1.4.2 scipy==1.13.0 semantic-version==2.10.0 sentence-transformers==2.7.0 sentencepiece==0.2.0 shellingham==1.5.4 six==1.16.0 sniffio==1.3.1 SQLAlchemy==2.0.29 sqlitedict==2.1.0 starlette==0.37.2 sympy==1.12.1rc1 tabledata==1.3.3 tabulate==0.9.0 tbb==2021.12.0 tcolorpy==0.1.4 tenacity==8.2.3 threadpoolctl==3.4.0 tokenizers==0.15.2 tomlkit==0.12.0 toolz==0.12.1 torch==2.1.0a0+cxx11.abi torchvision==0.16.0a0+cxx11.abi tqdm==4.66.2 tqdm-multiprocess==0.0.11 transformers==4.36.2 typepy==1.3.2 typer==0.12.3 typing-inspect==0.9.0 typing_extensions==4.11.0 tzdata==2024.1 urllib3==2.2.1 uvicorn==0.29.0 websockets==11.0.3 xxhash==3.4.1 yarl==1.9.4 zstandard==0.22.0

qiuxin2012 commented 2 months ago

It's still pip list, can you give the result of https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/scripts/env-check.bat

aoke79 commented 2 months ago

Python 3.11.9

transformers=4.36.2

torch=2.1.0a0+cxx11.abi

Name: ipex-llm Version: 2.1.0b20240707 Summary: Large Language Model Develop Toolkit Home-page: https://github.com/intel-analytics/ipex-llm Author: BigDL Authors Author-email: bigdl-user-group@googlegroups.com License: Apache License, Version 2.0 Location: C:\ProgramData\anaconda3\envs\ipex_llm_for_xpu\Lib\site-packages Requires: Required-by:

C:\ProgramData\anaconda3\envs\ipex_llm_for_xpu\Lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'C:\ProgramData\anaconda3\envs\ipex_llm_for_xpu\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpeg or libpng installed before building torchvision from source? warn( ipex=2.1.10+xpu

Total Memory: 31.725 GB

Chip 0 Memory: 4 GB | Speed: 7467 MHz Chip 1 Memory: 4 GB | Speed: 7467 MHz Chip 2 Memory: 4 GB | Speed: 7467 MHz Chip 3 Memory: 4 GB | Speed: 7467 MHz Chip 4 Memory: 4 GB | Speed: 7467 MHz Chip 5 Memory: 4 GB | Speed: 7467 MHz Chip 6 Memory: 4 GB | Speed: 7467 MHz Chip 7 Memory: 4 GB | Speed: 7467 MHz

CPU Manufacturer: GenuineIntel CPU MaxClockSpeed: 2500 CPU Name: Intel(R) Core(TM) Ultra 9 185H CPU NumberOfCores: 16 CPU NumberOfLogicalProcessors: 22

GPU 0: Intel(R) Arc(TM) Graphics Driver Version: 31.0.101.5333

System Information

Host Name: DESKTOP-L44TD4M OS Name: Microsoft Windows 11 Pro OS Version: 10.0.22631 N/A Build 22631 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free Registered Owner: wt Registered Organization: Product ID: 00330-80000-00000-AA523 Original Install Date: 10/9/2023, 2:16:42 PM System Boot Time: 6/25/2024, 10:07:45 AM System Manufacturer: Intel Corporation System Model: Meteor Lake Client Platform System Type: x64-based PC Processor(s): 1 Processor(s) Installed. 01: Intel64 Family 6 Model 170 Stepping 4 GenuineIntel ~2300 Mhz BIOS Version: Phoenix Technologies Ltd AMIB022A_2R7500, 10/10/2023 Windows Directory: C:\windows System Directory: C:\windows\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi Total Physical Memory: 32,487 MB Available Physical Memory: 23,756 MB Virtual Memory: Max Size: 58,451 MB Virtual Memory: Available: 48,835 MB Virtual Memory: In Use: 9,616 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \DESKTOP-L44TD4M Hotfix(s): 5 Hotfix(s) Installed.

                       [02]: KB5012170
                       [03]: KB5027397
                       [04]: KB5039212
                       [05]: KB5037959

Network Card(s): 1 NIC(s) Installed. 01: Intel(R) Wi-Fi 6E AX211 160MHz Connection Name: Wi-Fi DHCP Enabled: Yes DHCP Server: 1.1.1.1 IP address(es)

                             [02]: fe80::40c4:805a:8c32:2a5d

Hyper-V Requirements: VM Monitor Mode Extensions: Yes Virtualization Enabled In Firmware: Yes Second Level Address Translation: Yes Data Execution Prevention Available: Yes

+-----------+--------------------------------------------------------------------------------------+ | Device ID | Device Information | +-----------+--------------------------------------------------------------------------------------+ | 0 | Device Name: Intel(R) Arc(TM) Graphics | | | Vendor Name: Intel(R) Corporation | | | UUID: 00000000-0000-0200-0000-00087d558086 | | | PCI BDF Address: 0000:00:02.0 | +-----------+--------------------------------------------------------------------------------------+

Oscilloscope98 commented 2 months ago

Hi @aoke79 @Edward-Lin ,

You may have a try on the following steps before running the all-in-one benchmark again:

  1. Use the latest all-in-one benchmark
  2. pip uninstall bigdl-core-xe-esimd-21 in your python env
  3. Set env variable BIGDL_IMPORT_IPEX=1

Please let us know for any further problems :)

aoke79 commented 2 months ago

it works, thank very much!