TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len'

[Kushalamummigatti]

Please check that this issue hasn't been reported before.

• [X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Model training should run when the command accelerate launch -m axolotl.cli.train test.yaml is used.

Current behaviour

Throwing the error as below return forward_call(*args, kwargs) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward output = module._old_forward(*args, *kwargs) File "/home/aion/axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py", line 442, in flashattn_forward cos, sin = self.rotary_emb( File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, **kwargs) TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len' 0%| | 0/9 [00:01<?, ?it/s] Traceback (most recent call last): File "/opt/conda/envs/axolotl/bin/accelerate", line 8, in sys.exit(main()) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1023, in launch_command simple_launcher(args) File "/opt/conda/envs/axolotl/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/opt/conda/envs/axolotl/bin/python3.10', '-m', 'axolotl.cli.train', 'test.yaml']' returned non-zero exit status 1.

Steps to reproduce

Run the train command with the yaml file provided

Config yaml

base_model: codellama/CodeLlama-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: CodeLlamaTokenizer
is_llama_derived_model: true

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
  - path: summies.jsonl
    ds_type: json
    type: alpaca
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out/summies

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"

Possible solution

No response

Which Operating Systems are you using?

• [X] Linux
• [ ] macOS
• [ ] Windows

Python Version

3.10

axolotl branch-commit

OpenAccess-AI-Collective / axolotl

Acknowledgements

• [X] My issue title is concise, descriptive, and in title casing.
• [X] I have searched the existing issues to make sure this bug has not been reported yet.
• [X] I am using the latest version of axolotl.
• [X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

[imchkkim]

I had same issue here. Set flash_attention: false and got working

[e-p-armstrong]

Setting flash attention false fixed it for me too, thanks @imchkkim

[xiechengmude]

well. we cant use flash attn?

[chiragjn]

Going back to commit 40a88e8c4a2f32b63df0fe2079f7acfe73329273 on main should work

[Kushalamummigatti]

I had same issue here. Set flash_attention: false and got working

Yes this is the only solution. Thanks

[Gryphe]

Reverting to commit 40a88e8c4a2f32b63df0fe2079f7acfe73329273 and re-running pip3 install -e '.[flash-attn,deepspeed]' to downgrade Transformers/Accelerate resolved the issue for me. Only reverting to the mentioned commit wasn't enough to solve it for me.

Annoying, but better then disabling flash attention.

[NanoCode012]

Hello @Gryphe @chiragjn , thank you for pointing out that commit. Can anyone confirm this helped them too?

[BMPixel]

Downgrading transformers to version 4.38 or lower will solve the issue.

pip install transformers==4.38

This error occurs because in transformers version 4.39, the seq_len argument in the LlamaRotaryEmbedding.forward() function was deprecated. You can see the change at https://github.com/huggingface/transformers/commit/ffe60fdcd60c17c3f216694160c2521da90f984c#diff-06392bad3b9e97be9ade60d4ac46f73b6809388f4d507c2ba1384ab872711c51.

In the file axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py, line 290 calls LlamaRotaryEmbedding.forward() with the argument seq_len=kv_seq_len, which is no longer supported in the updated version of transformers.

[NanoCode012]

Will close this thanks to @BMPixel 's PR #1443

[kkk935208447]

Downgrading transformers to version 4.38 or lower will solve the issue.

pip install transformers==4.38

This error occurs because in transformers version 4.39, the seq_len argument in the LlamaRotaryEmbedding.forward() function was deprecated. You can see the change at huggingface/transformers@ffe60fd#diff-06392bad3b9e97be9ade60d4ac46f73b6809388f4d507c2ba1384ab872711c51.

In the file axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py, line 290 calls LlamaRotaryEmbedding.forward() with the argument seq_len=kv_seq_len, which is no longer supported in the updated version of transformers.

thanks, remove param "seq_len", it work