Closed longmalongma closed 6 months ago
I have not tried on the 24GB GPU before. But I think you can reduce the batch_size while increasing the accum_grad_iters and keeping the total_batch_size the same, so that less gpu memory is needed. For example, in this https://github.com/boheumd/MA-LMM/blob/main/run_scripts/breakfast/train.sh#L18
Ok, thanks for your reply! I see!
Thank you for your work, I only have 8 x 4090 (24gb) gpu, is this resource enough to reproduce your code?