Closed yeonju7kim closed 10 months ago
Hi! What does "there are 2 optimizers" mean? By design there should only be one.
https://github.com/Alpha-VLLM/LLaMA2-Accessory/issues/82
Sorry. I mean two FP32 AdamW momentums here. Does the FusedAdam consume twice memory? If I use AdamW instead of FusedAdam, can we reduce the memory usage?
These two state variables are required by the Adam algorithm per se, and is irrelevant to the exact implementation (Apex FusedAdam / torch AdamW). So you may not be able to save GPU memory from this aspect. If you could provide the number and type of GPUs you use, we might be able to offer you some more suggestions.
Thank you. My GPU is 3 A6000, 40GB. I trained my task with llama_adapter, but it didn't adapt to my task. Do you have any suggestion?
Full-parameter finetuning a 7B model is pretty hard given your hardware. Specifically, it involves 28GB memory for parameters, 28*2GB memory for optimizer states, and 28GB memory for gradients. When trained on 3 GPUs, the memory load on each GPU would be approximately 28 + 283/3 = 56G when using SDP, and more than (28 4) / 3 = 37GB when using FSDP, without considering the cost for intermediate results.
The following are my suggestions:
Thank you so much for your help. It really helps me. I should try PEFT techniques.
Hi! Thank you for developing this wonderful work.
I want to fully finetuning llama, but because the model has 2 optimizers, it is hard to load my model on the GPU. I want to know why there are 2 optimizers and if it is okay with only 1 optimizer. Thank you.