Open js333031 opened 1 month ago
Hey, I fixed the same error by upgrading my transformers
library to the latest version (4.43.2)
python -m pip install --upgrade transformers
Hey, I fixed the same error by upgrading my
transformers
library to the latest version (4.43.2)
python -m pip install --upgrade transformers
I did upgrade transformers. Getting a message finished successfully, but not seeing any output.
Command run - python run.py -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --prompt "why is the sky blue" --token-latency
Error: torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:198: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:199: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. torch.tensor(rope_type).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:202: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! self.max_seq_len_cached = max_seq_len_cached.item() /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/models.py:305: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! and hidden_states.size(1) != 1 ipex.llm.optimize has set the optimized or quantization model for model.generate() LLM RUNTIME INFO: Finished successfully.
@jianan-gu
I too tried the transformers lib update and got similar result as Rajiv above
Hey, I fixed the same error by upgrading my
transformers
library to the latest version (4.43.2)python -m pip install --upgrade transformers
I did upgrade transformers. Getting a message finished successfully, but not seeing any output.
Command run - python run.py -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --prompt "why is the sky blue" --token-latency
Error: torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:198: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:199: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. torch.tensor(rope_type).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:202: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! self.max_seq_len_cached = max_seq_len_cached.item() /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/models.py:305: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! and hidden_states.size(1) != 1 ipex.llm.optimize has set the optimized or quantization model for model.generate() LLM RUNTIME INFO: Finished successfully.
@mandalrajiv Missing "--benchmark" in your command, as mentioned in doc https://intel.github.io/intel-extension-for-pytorch/llm/llama3_1/cpu/#id1
@blzheng is working on the fix.
Hey, I fixed the same error by upgrading my
transformers
library to the latest version (4.43.2)python -m pip install --upgrade transformers
I did upgrade transformers. Getting a message finished successfully, but not seeing any output. Command run - python run.py -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --prompt "why is the sky blue" --token-latency Error: torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:198: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). torch.tensor(seq_len).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:199: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. torch.tensor(rope_type).contiguous(), /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/fusions/mha_fusion.py:202: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! self.max_seq_len_cached = max_seq_len_cached.item() /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/cpu/fusions/mha_fusion.py:234: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requiresgrad(True), rather than torch.tensor(sourceTensor). seq_info = torch.tensor( /usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/transformers/models/reference/models.py:305: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! and hidden_states.size(1) != 1 ipex.llm.optimize has set the optimized or quantization model for model.generate() LLM RUNTIME INFO: Finished successfully.
@mandalrajiv Missing "--benchmark" in your command, as mentioned in doc https://intel.github.io/intel-extension-for-pytorch/llm/llama3_1/cpu/#id1
@jianan-gu - I was not using the benchmark option. I was running the program based on an input prompt.
Ran the run.py program with a prompt, as well as the benchmark. The benchmark ran fine. But running with the inout prompt had the same outcome as before. Shows “finished successfully”, but no output is generated.
python run.py -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --prompt "why is the sky blue" --token-latency
deepspeed --bind_cores_to_rank run.py --benchmark -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --greedy --input-tokens 1024 --autotp --shard-model
Ran the run.py program with a prompt, as well as the benchmark. The benchmark ran fine. But running with the inout prompt had the same outcome as before. Shows “finished successfully”, but no output is generated.
Running with an input prompt
python run.py -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --prompt "why is the sky blue" --token-latency
Benchmark
deepspeed --bind_cores_to_rank run.py --benchmark -m meta-llama/Meta-Llama-3.1-8B-Instruct --dtype bfloat16 --ipex --greedy --input-tokens 1024 --autotp --shard-model
@mandalrajiv I did not get your point. What's the problem now? Could you show some instructions and details?
Thanks!
Describe the issue
Installed conda-forge env based llama3.1 setup per following instructions.
Encountering the error below, which seems to be similar to one discussed here