Closed srelbo closed 2 years ago
@zphang Would you be able to help us get this working? Thanks 🙏
I am using my own custom generate function and didn't have the invalid values problem, but I was also having issues with nonsensical generations that might possibly be related. I found that this was due to a mistake with caching the previous attention states on line 146
of modeling_gpt_neox.py
I think present = None if use_cache else (key, value)
should be present = (key,value) if use_cache else None
Changing that fixed my issue. You may also be able to set use_cache=False
as an argument to model.generate
, although that will give slightly slower generation.
Hi, sorry I've been busy. I should be able to take a look at this this weekend.
Thank you @benkrause and @zphang !
Hi @benkrause @zphang I have same issue. Can you please help me to resolve it ?
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Traceback (most recent call last):
File "app.py", line 15, in <module>
use_cache=False
File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1330, in generate
**model_kwargs,
File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1975, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Hi @benkrause @zphang I have same issue. Can you please help me to resolve it ?
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results. Setting `pad_token_id` to `eos_token_id`:2 for open-end generation. Traceback (most recent call last): File "app.py", line 15, in <module> use_cache=False File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1330, in generate **model_kwargs, File "/opt/conda/lib/python3.7/site-packages/transformers/generation_utils.py", line 1975, in sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
I'm also experiencing this issue. Have you already solved it?
System Info
Who can help?
@patil-suraj
I have been trying to using Huggingface GPTNeoX models to generate text. However even the basic example use case fails both on
CPU
andGPU
.Fails with
Similar failure happen when running on CUDA as well.
Based on -- https://huggingface.co/docs/transformers/main/en/model_doc/gpt_neox
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior