search

rmihaylov / falcontune

Tune any FALCON in 4-bit
Apache License 2.0
468 stars 51 forks source link

finetune on 4096 context length #16

Open MohamedAliRashad opened 1 year ago

MohamedAliRashad commented 1 year ago

How to finetune Falcon-7B-Instruct on input or outputs of 4096 context length ? how much VRAM i will need ?

yuhai-china commented 1 year ago

Does it work to set --cutoff_len=4096?

richardburleigh commented 1 year ago

Yes, I'm currently fine-tuning the 7B instruct model with 4096 and it's working fine.

Although I'm making the assumption that there's no truncation elsewhere in the code.