search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.68k stars 1.06k forks source link

Cannot attend to 9904, block size is only 4096 #1717

Closed starjob42 closed 1 month ago

starjob42 commented 2 months ago

Hi,

Thank you for awesome library!

I am using litgpt version 0.4.11 Currently I am using Phi-3.5-mini-instruct to finetune using lora. Even though I set --train.max_seq_length 10000 I still get 

ite-packages/litgpt/model.py", line 47, in max_seq_length
    raise ValueError(f"Cannot attend to {value}, block size is only {self.config.block_size}")
ValueError: Cannot attend to 9904, block size is only 4096

Here is the command that I am using to run:

litgpt finetune_lora microsoft/Phi-3.5-mini-instruct --data JSON --data.json_path my_data.json --data.val_split_fraction 0.1 --out_dir out/custom-model --train.max_seq_length 10000 --precision bf16-true --quantize bnb.nf4-dq # execute the litgpt command

rasbt commented 2 months ago

Thanks for the feedback. I think that's because this model is the 4k model. There is a 128k context length Phi 3.5 model, but that's not supported yet since it would require LongRoPE, which is not implemented, yet.