Can't detect GPU devices

[mudler]

Describe the bug

Context: I'm the author of LocalAI, and I'm trying to bring diffusers and transformers support to it ( https://github.com/mudler/LocalAI/pull/1746 ).

I'm starting by following the documentation in https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu&version=v2.1.10%2Bxpu , however It seems after successfully installing with conda all the dependencies, running the "Sanity" test I cannot find the devices in my system.

I have 2 Intel Arc A770, but when running:

python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"

The result is just:

2.1.0a0+cxx11.abi                                                                                                                          
2.1.10+xpu           

By printing torch.xpu.device_count(), it returns 0.

 cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

My user is in the video/render group:

uid=1000(mudler) gid=1000(mudler) groups=1000(mudler),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),109(render),110(lxd)                                                                                                                     

Running conda install is successfull, indeed seems I have all the packages:

conda install intel-extension-for-pytorch=2.1.10 pytorch=2.1.0 -c intel -c conda-forge

Channels:                                                                                                                                                                                                                                    
 - intel                                                                                                                                                                                                                                     
 - conda-forge                                                                                                                                                                                                                               
 - defaults                                                                                                                                                                                                                                  
Platform: linux-64                                                                                                                                                                                                                           
Collecting package metadata (repodata.json): done                                                                                                                                                                                            
Solving environment: done                                                                                                                                                                                                                    

# All requested packages already installed.       

system dependencies are there, indeed, I can run llama.cpp just fine and offloading everything to the GPU:

sudo apt install -y intel-oneapi-dpcpp-cpp-2024.0 intel-oneapi-mkl-devel=2024.0.0-49656                                                                                                                                                                                                                                                                                                                           Reading package lists... Done                                                                                                                                                                                                                Building dependency tree... Done                                                                                                                                                                                                             Reading state information... Done                                                                                                                                                                                                            intel-oneapi-mkl-devel is already the newest version (2024.0.0-49656).                                                                                                                                                                       
intel-oneapi-mkl-devel set to manually installed.                                                                                                                                                                                            
intel-oneapi-dpcpp-cpp-2024.0 is already the newest version (2024.0.2-49895).                                                                                                                                                                
intel-oneapi-dpcpp-cpp-2024.0 set to manually installed.                                                                                                                                                                                     
0 upgraded, 0 newly installed, 0 to remove and 64 not upgraded.                 

Since I am able to run llama.cpp within this host successfully (also via containers and kubernetes) I'm suspecting is somehow the python environment that cannot detect the devices.

Any help and hint would be greatly appreciated, thanks!

Versions

Collecting environment information...                  
PyTorch version: 2.1.0a0+cxx11.abi                     
PyTorch CXX11 ABI: Yes               
IPEX version: 2.1.10+xpu                
IPEX commit: a12f9f650                          
Build type: Release                             

OS: Ubuntu 22.04.3 LTS (x86_64)                 
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: N/A                              
IGC version: 2024.0.2 (2024.0.2.20231213)       
CMake version: version 3.22.1                                                                                         
Libc version: glibc-2.35                                                                                              

Python version: 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] (64-bit runtime)                                                                                                                                                           
Python platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
Is XPU available: False                         
DPCPP runtime version: 2024.0
MKL version: 2024.0            
GPU models and configuration:                 

Intel OpenCL ICD version: 23.43.27642.40-803~22.04
Level Zero version: 1.3.27642.40-803~22.04                                                                            

CPU:                                                                                                                  
Architecture:                       x86_64   
CPU op-mode(s):                     32-bit, 64-bit              
Byte Order:                         Little Endian
CPU(s):                             16
On-line CPU(s) list:                0-15
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen 7 5700G with Radeon Graphics
CPU family:                         25
Model:                              80
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
Stepping:                           0
Frequency boost:                    enabled
CPU max MHz:                        3800.0000
CPU min MHz:                        1400.0000
BogoMIPS:                           7586.08
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apici
d aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfct
r_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cq
m_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpcl
mulqdq rdpid overflow_recov succor smca fsrm               
Virtualization:                     AMD-V
L1d cache:                          256 KiB (8 instances)
L1i cache:                          256 KiB (8 instances)
L2 cache:                           4 MiB (8 instances)
L3 cache:                           16 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.1.10+xpu
[pip3] numpy==1.26.0
[pip3] torch==2.1.0a0+cxx11.abi
[conda] intel-extension-for-pytorch 2.1.10              py311_xpu_0    intel
[conda] numpy                     1.26.0                   pypi_0    pypi
[conda] pytorch                   2.1.0               py311_xpu_0    intel

Oddly enough, from the docker container it seems to detect the devices just fine:

docker run --rm -it --privileged --device=/dev/dri --ipc=host intel/intel-extension-for-pytorch:2.1.10-xpu bash 
ubuntu@95a16f7d8b36:/$ python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"
/usr/local/lib/python3.10/dist-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
2.1.0a0+cxx11.abi
2.1.10+xpu
[0]: _DeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512)
[1]: _DeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512)

[BismarckDD]

not sure the root cause, but i see the container(3.11) and host(3.10) have different python version in your env

besides, I see in the blog Llama2 inference it needs

source {ONEAPI_PATH}/setvars.sh

before execute the python cmd

python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"

btw, could you be able to inference llama2-13B model on 2 Arc A770 gpus? Thanks~

[intel-ravig]

@mudler - To start with, I would look into the drivers - specifically the UMD. I point to UMD because the intel-published docker container is picking up devices well.

Since level-zero is installed, can you run clinfo -l and post the output here?

You may have to install 'clinfo'.

If the outputs look well, then I concur with @BismarckDD , that likely the proper oneAPI environment variables were not sourced well, before running the python command. So - please try that as well.

[mudler]

@intel-ravig what I should look for UMD? I just installed the drivers as per the Intel docs I've linked, with the steps in the issue. This is a newly installed 22.04 LTS box.

Just for reference, here are the steps:

mudler@arc:~$ source activate diffusers
(diffusers) mudler@arc:~$ conda env list
# conda environments:
#
diffusers             *  /home/mudler/.conda/envs/diffusers
base                     /opt/conda

(diffusers) mudler@arc:~$ source /opt/intel/oneapi/setvars.sh 

:: initializing oneAPI environment ...
   -bash: BASH_VERSION = 5.1.16(1)-release
   args: Using "$@" for setvars.sh arguments: 
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: ipp -- latest
:: ippcp -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::
(diffusers) mudler@arc:~$ python3 -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"
2.1.0a0+cxx11.abi
2.1.10+xpu
(diffusers) mudler@arc:~$ clinfo -l
Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM)
 `-- Device #0: Intel(R) FPGA Emulation Device
Platform #1: Intel(R) OpenCL
 `-- Device #0: AMD Ryzen 7 5700G with Radeon Graphics         
Platform #2: Intel(R) OpenCL Graphics
 +-- Device #0: Intel(R) Arc(TM) A770 Graphics
 `-- Device #1: Intel(R) Arc(TM) A770 Graphics

[mudler]

to reiterate: on the same box with llama.cpp it all works fine and I can offload correctly to the GPUs. It just looks a problem with ipex

[intel-ravig]

@mudler - I am able to duplicate your issue on your conda environment. However, it runs fine on python venv and docker environments. I will discuss with engineering team for next course of action.

[mudler]

@mudler - I am able to duplicate your issue on your conda environment. However, it runs fine on python venv and docker environments. I will discuss with engineering team for next course of action.

thanks @intel-ravig !

for the time being in LocalAI I'll go with supporting it without conda - however conda support is much wanted as otherwise implementations become quite convoluted and harder to follow

[intel-ravig]

@mudler - I got in touch with the engineering team and got a solution. The conda environment issue is known and documented here.

https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/performance_tuning/known_issues.html

Problem: Number of dpcpp devices should be greater than zero.
    Cause: If you use Intel® Extension for PyTorch* in a conda environment, you might encounter this error. Conda also ships the libstdc++.so dynamic library file that may conflict with the one shipped in the OS.

    Solution: Export the libstdc++.so file path in the OS to an environment variable LD_PRELOAD.

I tried these steps: a. /sbin/ldconfig -p | grep stdc++ b. Pick the location of stdc++ for 64bit c. export LD_PRELOAD=<location of stdc++> d. 'conda activate' your environment e. source oneAPI variables f. Test the same python device command.

I was able to solve the issue with conda env.

Please check and let us know.

[ghost]

There currently exists a similar issue with intel/compute-runtime, however it's been observed on kernel 6.8 (kernel 6.7 apparently works normally). It might also manifest on your ancient kernel. Could this be the same cause? Here is the fix: https://github.com/intel/compute-runtime/issues/710#issuecomment-2002646557

[huiyan2021]

@mudler - I got in touch with the engineering team and got a solution. The conda environment issue is known and documented here.

https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/performance_tuning/known_issues.html

Problem: Number of dpcpp devices should be greater than zero.
    Cause: If you use Intel® Extension for PyTorch* in a conda environment, you might encounter this error. Conda also ships the libstdc++.so dynamic library file that may conflict with the one shipped in the OS.

    Solution: Export the libstdc++.so file path in the OS to an environment variable LD_PRELOAD.

I tried these steps: a. /sbin/ldconfig -p | grep stdc++ b. Pick the location of stdc++ for 64bit c. export LD_PRELOAD=<location of stdc++> d. 'conda activate' your environment e. source oneAPI variables f. Test the same python device command.

I was able to solve the issue with conda env.

Please check and let us know.

Hi @mudler Did this solution fix your issue?