Open Liqiandi opened 2 months ago
It seems that have successfully inferred several times, but this error occurred in subsequent inferring?
Which device do you use? Arc dicrete GPU on Windows? If you use Arc dicrete GPU on Windows, please disable intergrate GPU.
I have arc 770 and UHD 770. After disable UHD 770, this question has resolved. Thanks!
hi, I have successfully used the code below to test the speed of generating tokens when using qwen-7b under ipex.
`def main(model_dir = "Qwen/Qwen-7B-Chat"):
But when I use your official code (https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/dev/benchmark/all-in-one run.py)to test, such an error will occur. What is the reason please?
`Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:04<00:00, 1.82it/s] 2024-06-13 15:30:06,365 - INFO - Converting the current model to sym_int8 format...... current_device: 1
One day, Samantha stumbled upon a mysterious map that led to a hidden treasure model generate cost: 1.107833699999901 Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures, but her parents were hesitant to let her go on such an adventure.
One day, Samantha stumbled upon a mysterious map that led to a hidden treasure model generate cost: 1.343699099998048 Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures, but her parents were hesitant to let her go on such an adventure.
One day, Samantha stumbled upon a mysterious map that led to a hidden treasure model generate cost: 1.3271714000002248 Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures, but her parents were hesitant to let her go on such an adventure.
One day, Samantha stumbled upon a mysterious map that led to a hidden treasure onednn_verbose,info,oneDNN v3.3.0 (commit 887fb044ccd6308ed1780a3863c2c6f5772c94b3) onednn_verbose,info,cpu,runtime:threadpool,nthr:12 onednn_verbose,info,cpu,isa:Intel AVX2 with Intel DL Boost onednn_verbose,info,gpu,runtime:DPC++ onednn_verbose,info,gpu,engine,0,backend:Level Zero,name:Intel(R) UHD Graphics 770,driver_version:1.3.29283,binary_kernels:enabled onednn_verbose,info,gpu,engine,1,backend:Level Zero,name:Intel(R) Arc(TM) A770 Graphics,driver_version:1.3.29283,binary_kernels:enabled onednn_verbose,info,graph,backend,0:dnnl_backend onednn_verbose,info,experimental features are enabled onednn_verbose,info,use batch_normalization stats one pass is enabled onednn_verbose,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time onednn_verbose,graph,info,template:operation,engine,partition_id,partition_kind,op_names,data_formats,logical_tensors,fpmath_mode,backend,exec_time onednn_verbose,common,error,level_zero,errcode 1879048196 Traceback (most recent call last): File "D:\LQD\posefitness\posefitness\test\benchmark[run.py](http://run.py/)", line 1063, in run_transformer_int4_fp16_gpu_win output_ids = model.generate(input_ids, do_sample=False, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 1563, in generate return self.greedy_search( ^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 2385, in greedy_search outputs = self( ^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 533, in call return self.model(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1527, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\qwen_7b_chat\modeling_qwen.py", line 1043, in forward transformer_outputs = self.transformer( ^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1527, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\transformers\models[qwen.py](http://qwen.py/)", line 496, in qwen_model_forward outputs = block( ^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1527, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\qwen_7b_chat\modeling_qwen.py", line 610, in forward attn_outputs = self.attn( ^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1527, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\transformers\models[qwen.py](http://qwen.py/)", line 74, in qwen_attention_forward qkv = self.c_attn(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\torch\nn\modules[module.py](http://module.py/)", line 1527, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ProgramData\miniconda3\envs\posefit\Lib\site-packages\ipex_llm\transformers\low_bit_linear.py", line 734, in forward result = xe_linear.forward_new(x_2d, self.weight.data, self.weight.qtype, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: could not create a primitive`