Closed iamjanvijay closed 2 years ago
@patrickvonplaten Pinging you to get your input on this.
It seems -inf
are explicitely set by prefix_allowed_tokens_fn
and remove_invalid_values
doesn't remove float(-inf)
specifically.
However the script does seems to fail currently.
I added a PR containing the "fix" to accelerate things along, but given everything is ingrained in tests and other logits processors actively use float(-inf)
I am not sure this is the desired behavior.
Other options I consider viable:
float(-inf)
directly and use torch.finfo(scores.dtype).min
instead (we don't introduce infinities anymore so should solve it)float(-inf)
only before using torch.multinomialt
.I don't think that this is related in any way to the InfNanRemoveLogitsProcessor
processor. IMO, the reason for the error here is that in the 3rd generation step, all values of next_token_scores
are set to -inf
(I think) due to the prefix_allowed_tokens_fn
that you've added. This is not a bug IMO with transformers
, but with the prefix_allowed_tokens_fn
function as it should not set all values to -inf
.
A tip from my side @iamjanvijay would be to do the following. Create the PrefixConstrainedLogitsProcessor
object with your function and just play around with it locally (what happens at generation step 3) I think you'll see then that it sets all values to -inf
at some point which it shouldn't do
@patrickvonplaten @Narsil Thanks for your response. I was trying to check why this is happening. I found that if the restricted_vocab at any generation step only includes "\</s>" (end-of-sentence token) this error occurs. In other cases, the script doesn't encounter such an error. I'll try to look if all the elements at that generation step are set to -inf.
I'll close my PR in the meantime. We can reopen it if needed, but I tend to agree with @patrickvonplaten that having everything float(-inf)
can be considered a bug already.
@patrickvonplaten @Narsil Thanks for your response. I was trying to check why this is happening. I found that if the restricted_vocab at any generation step only includes "" (end-of-sentence token) this error occurs. In other cases, the script doesn't encounter such an error. I'll try to look if all the elements at that generation step are set to -inf.
Have you found what was causing the issue by any chance @iamjanvijay? I'm encountering the same issue while I'm using the generate function with BART, but I'm not using any prefix_allowed_tokens, and this error usually happens when I've been training the model for a while. Like @iamjanvijay said, I suspect something to do with cases where some tokens are masked or filtered, but I haven't really figured out where/why it's happening. I'd appreciate any pointers.
@mindojune - could you maybe open a new issue as it's not related to prefix_allowed_tokens_fn
?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
@mindojune Hi, I am facing the same problem as you and this error usually happens after I have trained the model for a while. And I am also using BART. Do you have any idea why this is happening or how to fix this error? Thank you a lot.
Hey @hongyuntw, the reason is that BART forces the second token to be this id https://huggingface.co/facebook/bart-large/blob/main/config.json#L27 . However if you use additionally something like prefix_allowed_tokens_fn
which might also not allow this id: https://huggingface.co/facebook/bart-large/blob/main/config.json#L27 => then all ids are set to -inf
in which case the model cannot generate anything anymore. To solve this I would probably set this config: https://huggingface.co/facebook/bart-large/blob/main/config.json#L27 to None
Was running into similar issues when using prefix_allowed_tokens_fn
in tandem with beam-search multinomial sampling, and realized the top_k
and top_p
args were sometimes preventing all the allowed tokens from being used, as they were outside those two tops. no_repeat_ngram_size
can have a similar effect.
Consider removing top_k
and top_p
if only allowing certain tokens is more important.
i also have this problem,but i dont know how to fix
Same I am also running into this issue, has there been any resolution for this ?
i try another way to avoid this problem,i was apply minigpt-4 in 3090,when i use v0 version weight,the problem happen,for result it ,i try many pytorch version ,but it dosent work.finally,i use a new model version v1.1,this problem also away.so,i think the problem is relative with model.for minigpt4,model decoding is relateted with fschat here
Did somebody solved this problem?
cc @gante for reference
@Chanwhistle have a look at this comment
If you believe this comment does not apply to you, then a reproducer of the issue will be needed 🤗
Environment info
transformers
version: 4.15.0Who can help
@patrickvonplaten @narsil
Information
Model I am using T5ForConditionalGeneration:
The problem arises when using my own modified scripts: Script to reproduce error is mentioned below.
The tasks I am working on is my own task or dataset: The task requires conditional generation from T5, in such a way, that the output vocabulary is restricted to a small set.
To reproduce
Expected behavior
No error.
Possible solution
The call function for class "InfNanRemoveLogitsProcessor" should include the following statement before returning "scores".