Open getao opened 4 months ago
cc @muellerzr @SunMarc
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
I would recommend you to try setting the liger kernels! 🤗 #32889
System Info
transformers=4.40.1 pytorch=2.2.1 deepspeed=0.14.1 accelerate=0.29.0
Who can help?
@pacman100 Hello, I tried fine-tuning LLMs using transformers' trainer with deepspeed zero-3 or fsdp. While I find it is very easy to use, it seems to cause more memory consumption (e.g., than litgpt's fsdp).
For example, when I fine-tune a 7b LLM (with bf16, flashattn, max context length=2048), with transformers' code (with zero-3 or fsdp), batch=2 (local) leads to OOM on a 8X40GB A100 node; while with litgpt, the (local) batch size can be set to 6 without OOM.
My fine-tuning script is as follows:
The training code is also straightforward:
Could I know whether there is something wrong with my code that leads to the additional memory cost?
Thank you
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
It can be easily run with my provided code and script.
Expected behavior
local batch size can be increased to 6 without OOM