Need Help About Long Context

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)

https://arxiv.org/abs/2403.13372

Apache License 2.0

35.31k stars 4.35k forks source link

Need Help About Long Context #5815

Open no-execution opened 1 month ago

no-execution commented 1 month ago

Reminder

[X] I have read the README and searched the existing issues.

System Info

llamafactory newest version

Reproduction

需要finetune 72B模型+32k context 用llamafactory总是爆显存，大家有什么好的办法吗？手头有100张左右的A100

Expected behavior

No response

Others

No response

XCF-Mike commented 1 month ago

Reminder

[x] I have read the README and searched the existing issues.

System Info

llamafactory newest version

Reproduction

需要finetune 72B模型+32k context 用llamafactory总是爆显存，大家有什么好的办法吗？手头有100张左右的A100

Expected behavior

No response

Others

No response

这个框架支持的unsloth和longlora试过吗，再就是deepspeed或者Xtuner的序列并行

hiyouga commented 1 month ago

try --enable_liger_kernel and --use_unsloth_gc parameter

mces89 commented 1 month ago

@hiyouga i tried these 2 options before, but they don't work for 70B + 32k length, do we have any plan to incorporate sequence parallism?