CFG and contrastive search not working for RWKV models in hf tf format

[KnutJaegersberg]

Describe the bug

I can load RWKV models, but I can't use CFG and also not contrastive search. By playing around, I get this error message. Sometimes there is no error message, it just doesn't do it. I can use both on other models.

I suspect in the case of CFG, it has not been implemented in HF transformers?

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

1. Load a HF transformers converted RWKV model, i.e. RWKV/rwkv-4-1b5-pile
2. Click contrastive search parameters option in parameters / use the CFG slider with values other than 1
3. Generate text

Screenshot

First attempt is contrastive search. No error message. Second is CFG. In either case only one of the two. Both combined don't work either.

Logs

File "/run/media/knut/HD/text-generation-webui/modules/callbacks.py", line 56, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/run/media/knut/HD/text-generation-webui/modules/text_generation.py", line 321, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/utils.py", line 1648, in generate
    return self.sample(
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/utils.py", line 2743, in sample
    next_token_scores = logits_processor(input_ids, next_token_logits)
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 97, in __call__
    scores = processor(input_ids, scores)
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 1655, in __call__
    logits = self.get_unconditional_logits(input_ids)
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 1640, in get_unconditional_logits
    out = self.model(
  File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: RwkvForCausalLM.forward() got an unexpected keyword argument 'past_key_values'

System Info

Linux, A6000

[Ph0rk0z]

How does RWKV work in transformers because I think it had it's own kernel. That may not support a second cache for CFG

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.