Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/opt/aws_neuronx_venv_transformers_neuronx/lib/python3.10/site-packages/transformers_neuronx/llama/model.py", line 240, in sample
result = sampling.sample_llama(
File "/opt/aws_neuronx_venv_transformers_neuronx/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/aws_neuronx_venv_transformers_neuronx/lib/python3.10/site-packages/transformers_neuronx/sampling.py", line 371, in sample_llama
return sample_loop_llama(
File "/opt/aws_neuronx_venv_transformers_neuronx/lib/python3.10/site-packages/transformers_neuronx/sampling.py", line 331, in sample_loop_llama
done_flags = torch.logical_or(done_flags, inputs == eos_token_id)
TypeError: logical_or(): argument 'other' (position 2) must be Tensor, not bool
My instance :
inf2.48xlarge
ami_image : Deep Learning AMI Neuron (Ubuntu 22.04)
While running the example given here : https://github.com/aws-neuron/aws-neuron-samples/blob/master/torch-neuronx/transformers-neuronx/inference/llama-3.1-8b-32k-sampling.ipynb , I am getting the following error when running the line
neuron_model.sample(input_ids, sequence_length=32768, top_k=10)
Error :
My instance :
inf2.48xlarge
Deep Learning AMI Neuron (Ubuntu 22.04)
ami-05d498302130f9036
Followed this doc : https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/neuron-setup/multiframework/multi-framework-ubuntu22-neuron-dlami.html and run the command
source /opt/aws_neuronx_venv_transformers_neuronx/bin/activate
Current Installed Libraries :