search

hao-ai-lab / LookaheadDecoding

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
https://arxiv.org/abs/2402.02057
Apache License 2.0
1.14k stars 67 forks source link

TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'lookahead' #59

Open zev123456 opened 6 months ago

zev123456 commented 6 months ago

run: USE_LADE=1 LOAD_LADE=1 python minimal.py result:

Traceback (most recent call last):
  File "/home/workspace/LookaheadDecoding/minimal.py", line 32, in <module>
    greedy_output = model.generate(**model_inputs, max_new_tokens=1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/transformers/generation/utils.py", line 1718, in generate
    return self.greedy_search(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/decoding.py", line 24, in greedy_search_proxy
    return jacobi_greedy_search_multilevel(self, chat=False, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/decoding.py", line 988, in jacobi_greedy_search_multilevel
    outputs = self.jforward_multilevel(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 1519, in jforward_multilevel
    outputs = self.model.LlamaModeljforward(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 1224, in LlamaModeljforward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 864, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn.forward(
                                                          ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'lookahead'

how solve this?

jivanph commented 5 months ago

I am having the same issue.

david-wei-01001 commented 5 months ago

I am having the same issue

david-wei-01001 commented 5 months ago

But mine is having an unexpected "padding_mask" TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'padding_mask'

jivanph commented 5 months ago

I solved this by downgrading my version of transformers. I believe version 4.36.2 (https://pypi.org/project/transformers/4.36.2/) worked for me.

david-wei-01001 commented 5 months ago

Yeah downgrade transformer to version 4.36.2 work for me as well So I guess this is resolved and the solution is to downgrade transformer to version 4.36.2

The team can close this haha Thank you very much jivanph