AttributeError: 'GenerationConfig' object has no attribute 'cache_implementation'

[qxpBlog]

System Info

python==3.8
fire==0.5.0
pydantic==1.10.6
torch==2.0.0
tqdm==4.65.0
sentencepiece==0.1.97
icetk==0.0.4 # ChatGLM
cpm_kernels==1.0.11  # ChatGLM
transformers==4.29.0
datasets==2.10.1
bitsandbytes==0.37.2  # Load 8bit
accelerate==0.17.1  # Load 8bit
numpy==1.24.2
peft==0.3.0
tiktoken==0.3.3  # OpenAI tokenizer
openai==0.27.6
einops==0.6.1
# ChatRWKV
rwkv==0.7.3
torchvision==0.15.1
safetensors==0.3.1
# LM-Eval
sacrebleu==2.3.1
protobuf==3.20.3
pycountry==22.3.5
rouge-score==0.1.2
pytablewriter==1.0.0

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

I met the following problem, how can i solve it

Traceback (most recent call last):
  File "/home/iotsc01/xinpengq/LLM-Pruner-main/instruct-eval-main/main.py", line 98, in <module>
    Fire(main)
  File "/home/iotsc01/anaconda3/envs/instruct-eval/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/iotsc01/anaconda3/envs/instruct-eval/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/iotsc01/anaconda3/envs/instruct-eval/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/iotsc01/xinpengq/LLM-Pruner-main/instruct-eval-main/main.py", line 30, in main
    score = task_fn(**kwargs)
  File "/home/iotsc01/xinpengq/LLM-Pruner-main/instruct-eval-main/bbh.py", line 83, in main
    result = evaluate(model, data, ntrain=ntrain)
  File "/home/iotsc01/xinpengq/LLM-Pruner-main/instruct-eval-main/bbh.py", line 67, in evaluate
    pred = model.run(prompt)
  File "/home/iotsc01/xinpengq/LLM-Pruner-main/instruct-eval-main/modeling.py", line 277, in run
    outputs = self.model.generate(
  File "/home/iotsc01/anaconda3/envs/instruct-eval/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/iotsc01/anaconda3/envs/instruct-eval/lib/python3.8/site-packages/transformers/generation/utils.py", line 1455, in generate
    if generation_config.cache_implementation in NEED_SETUP_CACHE_CLASSES_MAPPING and not model_kwargs.get(
AttributeError: 'GenerationConfig' object has no attribute 'cache_implementation'

Expected behavior

AttributeError: 'GenerationConfig' object has no attribute 'cache_implementation'

[amyeroberts]

Hi @qxpBlog, thanks for raising!

Please provide a minimal code reproducer that we can run to replicate the error.

cc @gante

[poganesh]

I am facing the same error too. The error arises when I call model.generate method which internally calls the transformers package, specifically from generation/utils.py. It arises when I call the model.generate method with the larger models (opt-1.3b, llama2) I am using. Does not arise with smaller models like opt-125m. I am using transformers==4.39.3

if generation_config.cache_implementation in NEED_SETUP_CACHE_CLASSES_MAPPING:
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^AttributeError: 'GenerationConfig' object has no attribute 'cache_implementation'

[rchynoweth]

I experienced this same error. After I added the cache_implementation config I then had an error related to the output_logits which is why its there too. Was able to fix it with the following code:

gen_conf = GenerationConfig.from_model_config(model.config)
gen_conf.cache_implementation='dynamic'
gen_conf.output_logits = False

# Model parameters can be tuned as desired.
outputs = model.generate(
        **inputs,
        do_sample=False,
        num_beams=5,
        max_length=256,
        min_length=1,
        top_p=0.9,
        repetition_penalty=1.5,
        length_penalty=1.0,
        temperature=1,
        generation_config=gen_conf,
      )

[gante]

Hi folks 👋

I need a short stand-alone script where the issue is clearly reproducible on any computer. Otherwise, it will be hard for me to find out what's wrong :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.