search

salesforce / progen

Official release of the ProGen models
BSD 3-Clause "New" or "Revised" License
627 stars 115 forks source link

fix bugs to enable key_value cache for generation #36

Open Kaffaljidhmah2 opened 1 year ago

Kaffaljidhmah2 commented 1 year ago

The prepare_inputs_for_generation() method is used at each decoding step for auto-regressive generation. The default name for the keyword argument of key-value caches is past_key_values instead of past. Renaming past to past_key_values fits the Huggingface Transformer interface and enables the key-value cache for the generation.

Please refer to https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L751.

salesforce-cla[bot] commented 1 year ago

Thanks for the contribution! Before we can merge this, we need @Kaffaljidhmah2 to sign the Salesforce Inc. Contributor License Agreement.

Kaffaljidhmah2 commented 1 year ago

I've signed the CLA. Please take a look if is possible.