search

meta-llama / codellama

Inference code for CodeLlama models
Other
16.06k stars 1.88k forks source link

Where is the attribute `past_key_values` #205

Closed XinyuShe closed 8 months ago

XinyuShe commented 9 months ago

I hope to use past_key_values for inferring longer texts, but even though I have set use_cache=True, I still cannot find it in the generated_tokens's attributes.

generated_tokens = model.generate(
                **model_input, 
                max_new_tokens=1600,
                use_cache=True
                )
jgehring commented 9 months ago

Hi @XinyuShe, I'm not sure what you're referring to since the repo doesn't feature a generate() function like the one you're describing.