Illegal Instruction (Core Dumped) when Running Model with CPU Docker Image

vauns commented 1 month ago

I encountered the following error when attempting to run the model Qwen/Qwen2.5-0.5B using the CPU version of the Docker image:

root@Ubuntu:/llm# python vllm_offline_inference.py 
WARNING 10-11 06:19:27 ray_utils.py:46] Failed to import Ray with ModuleNotFoundError("No module named 'ray'"). For distributed inference, please install Ray with `pip install ray`.
INFO 10-11 06:19:30 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='Qwen/Qwen2.5-0.5B', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=Qwen/Qwen2.5-0.5B)
WARNING 10-11 06:19:31 cpu_executor.py:143] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 10-11 06:19:31 selector.py:42] Using Torch SDPA backend.
INFO 10-11 06:19:32 weight_utils.py:199] Using model weights format ['*.safetensors']
2024-10-11 06:19:33,921 - INFO - Converting the current model to bf16 format......
2024-10-11 06:19:33,921 - INFO - Only HuggingFace Transformers models are currently supported for further optimizations
2024-10-11 06:19:34,974 - INFO - Only HuggingFace Transformers models are currently supported for further optimizations
INFO 10-11 06:19:34 cpu_executor.py:72] # CPU blocks: 21845
Processed prompts:   0%|                                                                                                                                                                                                                              | 0/4 [00:00<?, ?it/s]INFO 10-11 06:19:35 pynccl_utils.py:17] Failed to import NCCL library: NCCL only supports CUDA and ROCm backends.
INFO 10-11 06:19:35 pynccl_utils.py:18] It is expected if you are not running on NVIDIA GPUs.
Illegal instruction (core dumped)

I tested it on two environments, and the same error occurred on both:

1. Linux environment:

CPU

[root@Ubuntu~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             16
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Stepping:              2
CPU MHz:               2399.998
BogoMIPS:              4799.99
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K
NUMA node0 CPU(s):     0-7
NUMA node1 CPU(s):     8-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single ssbd rsb_ctxsw ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

Docker image: intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT

OS:


[root@Ubuntu ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"


- Docker version: 25.0.2

- Docker run command

docker run -itd \ --net=host \ --cpuset-cpus="0-15" \ --cpuset-mems="0" \ -v /root/.cache/huggingface/hub:/llm/models \ -e no_proxy=localhost,127.0.0.1 \ --memory="16G" \ --name=$CONTAINER_NAME \ --shm-size="8g" \ $DOCKER_IMAGE


- dmesg -T result

[Fri Oct 11 13:36:20 2024] traps: pt_main_thread[29610] trap invalid opcode ip:7f80a51e3844 sp:7ffc25109540 error:0 in _C.cpython-311-x86_64-linux-gnu.so[7f80a51d2000+43000] [Fri Oct 11 14:18:42 2024] traps: pt_main_thread[8717] trap invalid opcode ip:7fc52581e844 sp:7ffd2c3dab00 error:0 in _C.cpython-311-x86_64-linux-gnu.so[7fc52580d000+43000]


- collect_env result

root@Ubuntu # python collect_env.py Collecting environment information... WARNING 10-11 06:50:51 ray_utils.py:46] Failed to import Ray with ModuleNotFoundError("No module named 'ray'"). For distributed inference, please install Ray with pip install ray. PyTorch version: 2.3.0+cpu Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.4 Libc version: glibc-2.35

Python version: 3.11.10 (main, Sep 7 2024, 18:35:41) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-3.10.0-1160.108.1.el7.x86_64-x86_64-with-glibc2.35 Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz CPU family: 6 Model: 63 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 16 Stepping: 2 BogoMIPS: 4799.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single ssbd rsb_ctxsw ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities Hypervisor vendor: VMware Virtualization type: full L1d cache: 512 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 4 MiB (16 instances) L3 cache: 240 MiB (16 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Processor vulnerable Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; Load fences, usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full retpoline, IBPB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] intel-extension-for-pytorch==2.2.0 [pip3] numpy==1.26.4 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] torch==2.3.0+cpu [pip3] torchaudio==2.2.0+cpu [pip3] torchvision==0.17.0+cpu [pip3] triton==3.0.0 [conda] Could not collectROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.4.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect


### 2. Windows 11 WSL2 environment:
- CPU

[root@Ubuntu~]# lscpu root@DESKTOP-HQ9N5M2:~# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Vendor ID: GenuineIntel Model name: 12th Gen Intel(R) Core(TM) i7-12700H CPU family: 6 Model: 154 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 Stepping: 3 BogoMIPS: 5376.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fm a cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bm i2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities Virtualization features: Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full Caches (sum of all): L1d: 480 KiB (10 instances) L1i: 320 KiB (10 instances) L2: 12.5 MiB (10 instances) L3: 24 MiB (1 instance) Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Mitigation; Enhanced IBRS Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Srbds: Not affected Tsx async abort: Not affected

- Docker image:  intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT

- OS:

root@DESKTOP-HQ9N5M2:~# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy



- Docker version: 24.0.7

Could this be related to specific CPU instruction sets, or is there a workaround available? 

Any assistance would be appreciated. Thank you!

vauns commented 1 month ago

I have checked, and the Intel Xeon E5-2620 v3 does not support AVX-512 or AMX. My PC’s Intel Core i7-12700H also does not support AVX-512 or AMX.

When not using Docker, if I create a Python environment directly in my PC using Conda, then install ipex via pip and run the model, everything works fine. Can I understand this as the CPU-based Docker image potentially utilizing AVX-512 or AMX, while Windows only makes use of AVX2?

xiangyuT commented 1 month ago

Hi @vauns, I have tried the docker image intelanalytics/ipex-llm-serving-cpu:2.2.0-SNAPSHOT in our environment but I could not reproduce this issue with spr cpu.

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         52 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  192
  On-line CPU(s) list:   0-191
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Platinum 8468
    CPU family:          6
    Model:               143
    Thread(s) per core:  2
    Core(s) per socket:  48
    Socket(s):           2
    Stepping:            8
    CPU max MHz:         3800.0000
    CPU min MHz:         800.0000
    BogoMIPS:            4200.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_t
                         sc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg
                          fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 ca
                         t_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep 
                         bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsav
                         es cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg
                         _req vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemot
                         e movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities

vauns commented 1 month ago

Hi @xiangyuT,

I discovered that the CPU(Intel(R) Xeon(R) Platinum 8468) you're using supports AVX512 and AMX instruction sets (as indicated by the flag values). Based on this, I believe the issue on my side stems from the fact that the CPU of the Linux server I'm using does not support AVX512 and AMX.

My guess is based on the description of an Alibaba Cloud PyTorch-Intel image I saw earlier: ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/pytorch-intel:2.2.0.1-alinux3.2304 image requirements. The pytorch-intel image leverages Intel CPU's AVX512 and AMX instructions, which means it needs to run on hardware that supports these features.

After further research, including a GPT query, I confirmed that my CPU model does not support these two instruction sets.

Thank you for your time, and I appreciate any further insights or suggestions you might have.

vauns commented 1 month ago

I'm just puzzled that when not using Docker, running it on a Windows PC doesn't have this issue.

The runtime code in the Docker image seems to have some differences in instruction requirements compared to the code running on a Windows PC.

I haven't tested it on a Linux server without Docker yet, as my main focus is to use containers to meet my needs.

intel-analytics / ipex-llm

Illegal Instruction (Core Dumped) when Running Model with CPU Docker Image #12184

I encountered the following error when attempting to run the model Qwen/Qwen2.5-0.5B using the CPU version of the Docker image:

1. Linux environment: