ipex-llm[cpp] error: Sub-group size 8 is not supported on the device

[player1537]

I followed the instructions from https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html on a bare metal server from the Intel Dev Cloud, specifically this instance:

Intel® Max Series GPU (PVC) on 4th Gen Intel® Xeon® processors – 1100 series (8x)

I got these logs:

Test Command and Logs

```console $ $ ZE_ENABLE_LOADER_DEBUG_TRACE=1 SYCL_CACHE_PERSISTENT=1 ./main --n-gpu-layers 999 --n-predict 1024 --model ~/share/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf --ctx-size 4096 --ignore-eos --split-mode none --main-gpu 0 -f ~/opt/src prompt.txt Log start main: build = 1 (baa5868) main: built with Intel(R) oneAPI DPC++/C++ Compiler 2024.0.0 (2024.0.0.20231017) for x86_64-unknown-linux-gnu main: seed = 1716242491 llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /home/sdp/share/models/mistral-7b-instruct-v0.1.Q4_K_M.gguf (version GGUF V2) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = mistralai_mistral-7b-instruct-v0.1 llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 11: general.file_type u32 = 15 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V2 llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 8B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.07 GiB (4.83 BPW) llm_load_print_meta: general.name = mistralai_mistral-7b-instruct-v0.1 llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 2 '' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: LF token = 13 '<0x0A>' [SYCL] call ggml_init_sycl ggml_init_sycl: GGML_SYCL_DEBUG: 0 ggml_init_sycl: GGML_SYCL_F16: no ZE_LOADER_DEBUG_TRACE:Load Library of libze_intel_vpu.so.1 failed with libze_intel_vpu.so.1: cannot open shared object file: No such file or directory found 18 SYCL devices: | | | | |Max | |Max |Global | | | | | | |compute|Max work|sub |mem | | |ID| Device Type| Name|Version|units |group |group|size | Driver version| |--|-------------------|---------------------------------------|-------|-------|--------|-----|-------|---------------------| | 0| [level_zero:gpu:0]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 1| [level_zero:gpu:1]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 2| [level_zero:gpu:2]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 3| [level_zero:gpu:3]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 4| [level_zero:gpu:4]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 5| [level_zero:gpu:5]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 6| [level_zero:gpu:6]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 7| [level_zero:gpu:7]| Intel Data Center GPU Max 1100| 1.3| 448| 1024| 32| 51539M| 1.3.27191| | 8| [opencl:gpu:0]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| | 9| [opencl:gpu:1]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |10| [opencl:gpu:2]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |11| [opencl:gpu:3]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |12| [opencl:gpu:4]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |13| [opencl:gpu:5]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |14| [opencl:gpu:6]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |15| [opencl:gpu:7]| Intel Data Center GPU Max 1100| 3.0| 448| 1024| 32| 48946M| 23.35.27191.42| |16| [opencl:cpu:0]| Intel Xeon Platinum 8468V| 3.0| 192| 8192| 64|1081858M|2024.17.3.0.08_160000| |17| [opencl:acc:0]| Intel FPGA Emulation Device| 1.2| 192|67108864| 64|1081858M|2024.17.3.0.08_160000| ggml_backend_sycl_set_single_device: use single device: [0] use 1 SYCL GPUs: [0] with Max compute units:448 llm_load_tensors: ggml ctx size = 0.30 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: SYCL0 buffer size = 4095.05 MiB llm_load_tensors: CPU buffer size = 70.31 MiB .............................................................................................. llama_new_context_with_model: n_ctx = 4096 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: SYCL0 KV buffer size = 512.00 MiB llama_new_context_with_model: KV self size = 512.00 MiB, K (f16): 256.00 MiB, V (f16): 256.00 MiB llama_new_context_with_model: SYCL_Host output buffer size = 0.12 MiB llama_new_context_with_model: SYCL0 compute buffer size = 296.00 MiB llama_new_context_with_model: SYCL_Host compute buffer size = 16.01 MiB llama_new_context_with_model: graph nodes = 1062 llama_new_context_with_model: graph splits = 2 Sub-group size 8 is not supported on the device Exception caught at file:/home/runner/_work/llm.cpp/llm.cpp/llama-cpp-bigdl/ggml-sycl.cpp, line:15352, func:operator() SYCL error: CHECK_TRY_ERROR(op(src0, src1, dst, src0_dd_i, src1_ddf_i, src1_ddq_i, dst_dd_i, dev[i].row_low, dev[i].row_high, src1_ncols, src1_padded_col_size, stream)): Meet error in this line code! in function ggml_sycl_op_mul_mat at /home/runner/_work/llm.cpp/llm.cpp/llama-cpp-bigdl/ggml-sycl.cpp:15352 GGML_ASSERT: /home/runner/_work/llm.cpp/llm.cpp/llama-cpp-bigdl/ggml-sycl.cpp:3021: !"SYCL error" Could not attach to process. If your uid matches the uid of the target process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf ptrace: Inappropriate ioctl for device. No stack. The program is not being run. ```

I suspect the problem is related to the fact that I'm using a machine with 8 GPUs (given the log statement about a "sub-group size 8 is not supported on the device").

I was able to successfully compile and run the llama.cpp source code myself with no issues, so I believe the problem is related to how exactly the IPEX-LLM version of llama.cpp was compiled.

(P.S. When I compiled it myself, I used the 2024.0 version of the oneAPI compiler package)

Any help would be greatly appreciated!

[rnwang04]

Hi @player1537 , I have reproduced this error. We will try to fix it, and once it is done, will update here to let you know : )

[rnwang04]

Hi @player1537 , I have fixed this issue, maybe you can try it again with ipex-llm[cpp] >= 2.1.0b20240521 (which will be released tonight). By the way, if you have no special requirements for accuracy, we recommend you use Q4_0, which provides the fastest speed on PVC : )

[player1537]

Thank you for the quick update!

I just wanted to confirm that I was able to get IPEX-LLM working now on the aforementioned machine.

We're seeing ~2.2x the inference token/s with IPEX-LLM's llama.cpp as we got with my personally compiled llama.cpp! :)

[soulyet]

but we met same issue with IPEX-LLM v2.1.0b20240819. is there something wrong with my environment?

GPU: Intel Arc GPU

logs: llama.cpp_log.txt

[lhl]

FYI, I am getting this same error with my Xe2 iGPU (Arc Graphics 140V). Q4_0 works, but Q4_K_M gives me this error. I am using oneAPI Base Toolkit 2024.2.1 w/ llama.cpp (it won't run with 2025.0.0).