TypeError: Phi3Attention.forward() got an unexpected keyword argument 'phantom_position'

[TanaroSch]

Thank you for publishing this great model! I followed the conda env creation on the huggingface page and tried to run the demo.py. The demo worked using the 1.8B model. However, when trying to use the 3.8b version, I receive the following error. I'm running it in WSL 2.

Traceback (most recent call last):
  File "/mnt/c/AI/LLM-Vision/Phantom/Phantom/demo.py", line 41, in <module>
    generate_ids = model.generate(**_inputs, do_sample=False, max_new_tokens=256)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/AI/LLM-Vision/Phantom/Phantom/model/arch_3_8b/modeling_phantom.py", line 164, in generate
    outputs = self.language_model.generate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/transformers/generation/utils.py", line 1479, in generate
    return self.greedy_search(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/transformers/generation/utils.py", line 2340, in greedy_search
    outputs = self(
              ^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/AI/LLM-Vision/Phantom/Phantom/model/arch_3_8b/modeling_phi3.py", line 1361, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/AI/LLM-Vision/Phantom/Phantom/model/arch_3_8b/modeling_phi3.py", line 1237, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/phantom/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/AI/LLM-Vision/Phantom/Phantom/model/arch_3_8b/modeling_phi3.py", line 948, in forward
-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Phi3Attention.forward() got an unexpected keyword argument 'phantom_position'```

[ByungKwanLee]

What is your transformers version?

[TanaroSch]

My transformers version is 4.37.2.

[ByungKwanLee]

Thanks for reporting! I will try 3.8b definitely in next tomorrow!

[ByungKwanLee]

Sorry to late reply. I've solved this issue. This issue comes from attn_implementation (flash attention), and when I converted dev code into this official repo, it seemed to happen.

You can use the latest version of this code

[TanaroSch]

Thank you! It works now :)