qiuxin2012
commented
3 months ago Sorry, our benchmark utils can't support transformers >= 4.42, you can use the default 4.36.2 with pip install transformers==4.36.2.
Closed JianxiaoLuIntel closed 3 months ago
qiuxin2012
commented
3 months ago Sorry, our benchmark utils can't support transformers >= 4.42, you can use the default 4.36.2 with pip install transformers==4.36.2.
JianxiaoLuIntel
commented
3 months ago @qiuxin2012 Thanks, it works. But I still met other runtime errors during the benchmark running
(llm) C:\Users\test\Desktop\jianxiao\ipex-llm\python\llm\dev\benchmark\all-in-one>python run.py
C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
2024-07-08 10:27:58,669 - INFO - intel_extension_for_pytorch auto imported
C:\Users\test\miniforge3\envs\llm\Lib\site-packages\transformers\deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
warnings.warn(
Loading checkpoint shards: 100%|¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€| 2/2 [00:47<00:00, 23.50s/it]
2024-07-08 10:28:46,731 - INFO - Converting the current model to sym_int4 format......
>> loading of model costs 71.1368687999784s and 4.056640625GB
<class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
C:\Users\test\miniforge3\envs\llm\Lib\site-packages\transformers\generation\configuration_utils.py:389: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
C:\Users\test\miniforge3\envs\llm\Lib\site-packages\transformers\generation\configuration_utils.py:394: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
warnings.warn(
model generate cost: 8.207748699991498
<s><s> Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of becoming a famous singer, but her parents were not supportive of her dream. They wanted her to focus on her studies and get a "real" job
model generate cost: 4.333771500008879
<s><s> Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of becoming a famous singer, but her parents were not supportive of her dream. They wanted her to focus on her studies and get a "real" job
model generate cost: 4.360847300005844
<s><s> Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of becoming a famous singer, but her parents were not supportive of her dream. They wanted her to focus on her studies and get a "real" job
model generate cost: 4.343568299984327
<s><s> Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of becoming a famous singer, but her parents were not supportive of her dream. They wanted her to focus on her studies and get a "real" job
model generate cost: 26.794378199992934
<s><s> The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. It was as if the stories were trying to tell him something, to reveal a hidden truth that he needed to uncover.
He spent the rest of the day in the library, pouring over the books and taking notes on the ones that seemed most relevant. As the sun set and the library closed, he left with a newfound sense of purpose. He knew that he had a long road ahead of him, but he was determined to uncover the truth that lay hidden in the shadows of his mind.
[CHAPTER 5:
THE DARK FOREST](978
model generate cost: 26.41183210001327
<s><s> The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. It was as if the stories were trying to tell him something, to reveal a hidden truth that he needed to uncover.
He spent the rest of the day in the library, pouring over the books and taking notes on the ones that seemed most relevant. As the sun set and the library closed, he left with a newfound sense of purpose. He knew that he had a long road ahead of him, but he was determined to uncover the truth that lay hidden in the shadows of his mind.
[CHAPTER 5:
THE DARK FOREST](978
model generate cost: 26.295101800002158
<s><s> The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. It was as if the stories were trying to tell him something, to reveal a hidden truth that he needed to uncover.
He spent the rest of the day in the library, pouring over the books and taking notes on the ones that seemed most relevant. As the sun set and the library closed, he left with a newfound sense of purpose. He knew that he had a long road ahead of him, but he was determined to uncover the truth that lay hidden in the shadows of his mind.
[CHAPTER 5:
THE DARK FOREST](978
model generate cost: 26.28936269998667
<s><s> The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. It was as if the stories were trying to tell him something, to reveal a hidden truth that he needed to uncover.
He spent the rest of the day in the library, pouring over the books and taking notes on the ones that seemed most relevant. As the sun set and the library closed, he left with a newfound sense of purpose. He knew that he had a long road ahead of him, but he was determined to uncover the truth that lay hidden in the shadows of his mind.
[CHAPTER 5:
THE DARK FOREST](978
Exception in thread Thread-4 (run_model_in_thread):
Traceback (most recent call last):
File "C:\Users\test\miniforge3\envs\llm\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Users\test\miniforge3\envs\llm\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\test\Desktop\jianxiao\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 55, in run_model_in_thread
output_ids = model.generate(input_ids, do_sample=False, max_new_tokens=out_len,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 1563, in generate
return self.greedy_search(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 2385, in greedy_search
outputs = self(
^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\utils\benchmark_util.py", line 533, in __call__
return self.model(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 1181, in forward
outputs = self.model(
^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\transformers\models\llama.py", line 122, in llama_model_forward_4_36
return llama_model_forward_4_36_internal(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\transformers\models\llama.py", line 2775, in llama_model_forward_4_36_internal
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\transformers\models\llama.py", line 321, in llama_decoder_forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\transformers\models\llama.py", line 1537, in llama_attention_forward_4_38
return forward_function(
^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\ipex_llm\transformers\models\llama.py", line 2027, in llama_attention_forward_4_38_original
attn_output = attn_output.transpose(1, 2).contiguous()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)
Traceback (most recent call last):
File "C:\Users\test\Desktop\jianxiao\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 1871, in <module>
run_model(model, api, in_out_pairs, conf['local_model_hub'], conf['warm_up'], conf['num_trials'], conf['num_beams'],
File "C:\Users\test\Desktop\jianxiao\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 119, in run_model
result = run_transformer_int4_gpu(repo_id, local_model_hub, in_out_pairs, warm_up, num_trials, num_beams, low_bit, batch_size, cpu_embedding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\Desktop\jianxiao\ipex-llm\python\llm\dev\benchmark\all-in-one\run.py", line 555, in run_transformer_int4_gpu
model.to('cpu')
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\transformers\modeling_utils.py", line 2460, in to
return super().to(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1160, in to
return self._apply(convert)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
module._apply(fn)
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 833, in _apply
param_applied = fn(param)
^^^^^^^^^
File "C:\Users\test\miniforge3\envs\llm\Lib\site-packages\torch\nn\modules\module.py", line 1158, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)I found the solution in other issues,
set IPEX_LLM_LOW_MEM=1 will solve this, thanks.
I am trying to setup the benchmark following the instruction https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/benchmark_quickstart.md
I met runtime errors running ipex-llm/python/llm/dev/benchmark/all-in-one/run,py : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, xpu:0 and cpu! (when checking argument for argument test_elements in method wrapper_XPU_Tensor_Tensor_out_isin_out)
Device info
CPU: Intel Core i7-1360P GPU: Intel Iris Xe Graphics (driver version: 31.0.101.5594) RAM: 16GB OS: Windows 11 (version: 22631.3737)
config.yaml
I just copy the config from the doc, only modify the model path
model
Download the model from https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/tree/main
conda list
Log