[Windows] [Pyinstaller] Process exit when I use "Pyinstaller" to package, the executable cannot detect XPU.

[tjtanaa]

Describe the issue

I have no issue when running before packaging.

I encounter the same issue with pyinstaller compiled executable. The program exited within AutoModelForCausalLM without any error. I can observed that the progress bar of loading the weight is complete.

        self.model = AutoModelForCausalLM.from_pretrained(
            model_path, trust_remote_code=True
        ).to(self.device)

There is also this error when launching the intel_extension_for_pytorch.

intel_extension_for_pytorch\xpu\lazy_init.py when loading extension _C._initExtension() is causing the following error. _C is _C.cp310-win_amd64.pyd

 intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.)
  _C._initExtension()

[huiyan2021]

Hi @tjtanaa, could you share a reproducer step by step? Thanks!

[tjtanaa]

Hi @huiyan2021 . The following is the steps to reproduce the issue on Windows 11. Conda environment is required.

Requisite

1. Download a LLM model. e.g. meta-llama/Meta-Llama-3-8B-Instruct

Steps to Reproduce

1. Install embeddedllm.
2. git clone https://github.com/EmbeddedLLM/embeddedllm.git --branch ipex-llm-win-tj
3. cd embeddedllm
4. conda create -n ellm python=3.10 libuv; conda activate ellm
5. ELLM_TARGET_DEVICE='ipex' python setup.py develop
6. Go to C:\Users\<user>\anaconda3\envs\ellmifp\Lib\site-packages\ipex_llm\transformers\convert.py of your conda environment to change the following line 56: _IS_VLLM_AVAILABLE = None to _IS_VLLM_AVAILABLE = False.
7. Install PyInstaller: pip install pyinstaller==6.9.0.
8. Compile Windows Executable: pyinstaller .\ellm_api_server.spec.
9. You can find the executable in the dist\ellm_api_server.
10. Run the following in Windows .\ellm_api_server.exe --model_path 'C:\Users\path\to\meta-llama_Meta-Llama-3-8B-Instruct' --backend 'ipex' --device 'xpu'
11. Expected stdout logs:

    
    pydantic\_internal\_fields.py:160: UserWarning: Field "model_path" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = ('settings_',). The system is Windows. torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpeg or libpng installed before building torchvision from source? warn( intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.) _C._initExtension() 2024-07-15 17:48:09,901 - INFO - intel_extension_for_pytorch auto imported 2024-07-15 17:48:10.057 | INFO | embeddedllm.backend.ipex_engine:init:43 - Model Context Lenght: 8192 2024-07-15 17:48:10.058 | INFO | embeddedllm.backend.ipex_engine:init:46 - Attempt to load fast tokenizer Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 9.82it/s] 2024-07-15 17:48:11,547 - INFO - Converting the current model to sym_int4 format......

[huiyan2021]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

[tjtanaa]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

Hi, @huiyan2021 , could you give me some pointers as to which part of the code that is generates _C.cp310-win_amd64.pyd, so that in the meantime, I could try to look into it on my end.

[xiguiw]

Steps to Reproduce

1. Install embeddedllm.
2. git clone https://github.com/EmbeddedLLM/embeddedllm.git --branch ipex-llm-win-tj
3. cd embeddedllm
4. conda create -n ellm python=3.10 libuv; conda activate ellm
5. ELLM_TARGET_DEVICE='ipex' python setup.py develop
6. Go to C:\Users\<user>\anaconda3\envs\ellmifp\Lib\site-packages\ipex_llm\transformers\convert.py of your conda environment to change the following line 56: _IS_VLLM_AVAILABLE = None to _IS_VLLM_AVAILABLE = False.

@tjtanaa What's your environment to reproduce this issue. The Intel extension-for-pytorch version, oneAPI version, driver version? What's the ipex_llm is?

[xiguiw]

There is also this error when launching the intel_extension_for_pytorch.

intel_extension_for_pytorch\xpu\lazy_init.py when loading extension _C._initExtension() is causing the following error. _C is _C.cp310-win_amd64.pyd

 intel_extension_for_pytorch\xpu\lazy_init.py:80: UserWarning: XPU Device count is zero! (Triggered internally at C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127.)
  _C._initExtension()

@tjtanaa How did you install IPEX? What are these: "C:/Users/arc/ruijie/2.1_RC3/python310/frameworks.ai.pytorch.ipex-gpu/csrc/gpu/runtime/Device.cpp:127". I did not find any 'gpu' in IPEX source files (it's 'xpu'). And I did find out the file 'Device.cpp' in IPEX public source code. This is internal innersource code.

[jingxu10]

Hi @tjtanaa , We don't support PyInstaller and no bandwidth at the time being.

Hi, @huiyan2021 , could you give me some pointers as to which part of the code that is generates _C.cp310-win_amd64.pyd, so that in the meantime, I could try to look into it on my end.

@ashokei @min-jean-cho any insights?

[min-jean-cho]

UserWarning: XPU Device count is zero!

Could you please run command sycl-ls after activating oneAPI environment to verify your GPU device is being detected?

[tjtanaa]

@xiguiw I installed ipex-llm from pypi package. The steps is based on this Ipex-llm: Install Windows GPU which runs the command pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

When installing embeddedllm it will run this same command pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/.

The following is my python environment packages

``` Package Version Editable project location --------------------------- ------------------ ------------------------------------------- accelerate 0.23.0 aiohttp 3.9.5 aiosignal 1.3.1 altgraph 0.17.4 annotated-types 0.7.0 anyio 4.4.0 async-timeout 4.0.3 attrs 23.2.0 bigdl-core-xe-21 2.5.0b20240714 bigdl-core-xe-addons-21 2.5.0b20240714 bigdl-core-xe-batch-21 2.5.0b20240714 certifi 2024.7.4 charset-normalizer 3.3.2 click 8.1.7 colorama 0.4.6 datasets 2.20.0 dill 0.3.8 distro 1.9.0 docstring_parser 0.16 dpcpp-cpp-rt 2024.0.2 embeddedllm 0.0.1+ipex C:\Users\hpintel\VDrive\git\embeddedllm-ifp exceptiongroup 1.2.2 fastapi 0.110.3 filelock 3.15.4 filetype 1.2.0 frozenlist 1.4.1 fsspec 2024.5.0 gunicorn 22.0.0 h11 0.14.0 httpcore 1.0.5 httpx 0.27.0 huggingface-hub 0.23.4 idna 3.7 inquirerpy 0.3.4 intel-cmplr-lib-rt 2024.0.2 intel-cmplr-lic-rt 2024.0.2 intel-extension-for-pytorch 2.1.10+xpu intel-opencl-rt 2024.0.2 intel-openmp 2024.0.2 ipex-llm 2.1.0b20240714 Jinja2 3.1.4 loguru 0.7.2 markdown-it-py 3.0.0 MarkupSafe 2.1.5 mdurl 0.1.2 mkl 2024.0.0 mkl-dpcpp 2024.0.0 mpmath 1.3.0 multidict 6.0.5 multiprocess 0.70.16 networkx 3.3 numpy 1.26.4 onednn 2024.0.0 onemkl-sycl-blas 2024.0.0 onemkl-sycl-datafitting 2024.0.0 onemkl-sycl-dft 2024.0.0 onemkl-sycl-lapack 2024.0.0 onemkl-sycl-rng 2024.0.0 onemkl-sycl-sparse 2024.0.0 onemkl-sycl-stats 2024.0.0 onemkl-sycl-vm 2024.0.0 openai 1.35.13 packaging 24.1 pandas 2.2.2 pefile 2023.2.7 pfzy 0.3.4 pillow 10.3.0 pip 24.0 prompt_toolkit 3.0.47 protobuf 5.28.0rc1 psutil 6.0.0 py-cpuinfo 9.0.0 pyarrow 16.1.0 pyarrow-hotfix 0.6 pydantic 2.7.4 pydantic_core 2.18.4 pydantic-settings 2.3.4 Pygments 2.18.0 pyinstaller 6.9.0 pyinstaller-hooks-contrib 2024.7 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 pytz 2024.1 pywin32-ctypes 0.2.2 PyYAML 6.0.1 regex 2024.5.15 requests 2.32.3 rich 13.7.1 safetensors 0.4.3 sentencepiece 0.2.0 setuptools 69.5.1 shtab 1.7.1 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.13.0 tabulate 0.9.0 tbb 2021.13.0 tokenizers 0.15.2 torch 2.1.0a0+cxx11.abi torchvision 0.16.0a0+cxx11.abi tqdm 4.66.4 transformers 4.36.2 trl 0.9.6 typing_extensions 4.12.2 tyro 0.8.5 tzdata 2024.1 urllib3 2.2.2 uvicorn 0.30.1 wcwidth 0.2.13 wheel 0.43.0 win32-setctime 1.1.0 xxhash 3.4.1 yarl 1.9.4 ```

[tjtanaa]

UserWarning: XPU Device count is zero!

Could you please run command sycl-ls after activating oneAPI environment to verify your GPU device is being detected?

@min-jean-cho I forgot to share my hardware setup. It is a Windows 11 PC with Hardware: Intel(R) Core(TM) Ultra 7 165H

The ipex-llm[xpu] supports the use of iGPU of Intel(R) Core(TM) Ultra 7 165H. The ipex-llm[xpu] is able to detect and utilized the iGPU when I run through python command. However, after packaging it into executable using pyInstaller, the executable cannot detect and utilized the iGPU and fail to run. After tracing the error, I found that the only different in log message is when loading the _C library of intel-extension-for-pytorch and it says XPU device is not found.

[MaksimDanilov]

@tjtanaa, Hello. I use pyinstaller once to make exe and succeed. You need to ensure that all dll including

you have in PATH or in your dist package. And of course, pyinstaller knows nothing about ipex package and its dependencies, so you need to do everything manually.

[tjtanaa]

@tjtanaa, Hello. I use pyinstaller once to make exe and succeed. You need to ensure that all dll including

you have in PATH or in your dist package. And of course, pyinstaller knows nothing about ipex package and its dependencies, so you need to do everything manually.

@MaksimDanilov Thank you so much for the great news. Can u share the steps to include all the dll? Did you install the OneAPI DPC++ Compiler and OneAPI Math Kernel Library (oneMKL) from source? I can't seem to find the steps to build ipex-llm from source. Could you point me to some directions to compile ipex-llm from source?

[MaksimDanilov]

@tjtanaa firstly, lets check if everything works fine. Try to launch your exe in console, but before that use Intel oneAPI setvars.bat as suggested at installation page. if you don't have oneAPI installed, then update PATH variable with installation path of anaconda packages, that correspond with above dependencies. If everything works fine, you are free-to-go to find all deps of your exe.

[tjtanaa]

@MaksimDanilov Thank you very much for the confidence and pointers. I have managed to compile a pyinstaller executable. I found that the latest Arc Driver is not compatible with ipex-llm. The engine will hang. So I downgraded and tested that the driver version 32.0.101.5762 is working for me.