Closed ValeKnappich closed 1 year ago
cc @gante
Hey @ValeKnappich 👋
Yeah, model_kwargs
needs to be added to _validate_model_kwargs
. I'm on it :)
Great, thanks :)
@gante @sgugger
The kwarg validation was only a superficial issue. In fact, now it does not throw an error anymore, however, the past_key_values
are still not passed on to the forward method. Looks like the prepare_inputs_for_generation
method is at the core of the problem:
def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, **model_kwargs):
input_shape = input_ids.shape
# if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
if attention_mask is None:
attention_mask = input_ids.new_ones(input_shape)
# cut decoder_input_ids if past is used
if past and past[0] is not None:
input_ids = input_ids[:, -1:]
return {"input_ids": input_ids, "attention_mask": attention_mask, "past_key_values": past}
Note that model_kwargs is simply swallowed here. I will create a PR shortly
@gante @ArthurZucker I think we should rename all occurrences of "past"
to "past_key_values"
in prepare_inputs_for_generation
and deprecate "past" if necessary.
"past"
was simply the name for the past key values states before we renamed everything to past_key_values
, so this is just a left-over.
Agreed
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
System Info
Python 3.7.13 transformers 4.22.2
Who can help?
@LysandreJik @patrickvonplaten
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
The
past_key_values
kwarg is not accepted when callingmodel.generate(..., past_key_values=pkv)
on aGPTNeoxForCausalLM
, even though themodel.forward
does accept this kwarg. It does seem to work fine with other model classes like GPT2.Minimal example to reproduce error:
Error message:
I checked the error location and located the bug ("transformers/generation_utils.py", line 862, in _validate_model_kwargs):
It first checks the args of
prepare_inputs_for_generation
and only adds the args offorward
to the accepted list if"kwargs"
is in the args ofprepare_inputs_for_generation
. However, contrary to GPT2, it only containsmodel_kwargs
instead ofkwargs
for GPTNeox.So either the GPTNeoX class should be adapted, or the _validate_model_kwargs method in generation_utils.py.
Expected behavior
generate
should be able to pass along all validmodel_kwargs