Closed SingL3 closed 12 months ago
Yes, I also noticed that our current trainer code / configurations don't work even for smaller models on single 80 GB GPUs. It would be great to get this analyzed/fixed.
@andreaskoepf I will take a look into this issue and try to fix some of them (I think there may be multiple reasons for this). If you have any clue or suggestion, please let me know and I would appreciate.
Previously, I have trained a pythia-6.9b using code here: dolly I can train with the below setting on 4xA100(80G) without GPU OOM:
with deepspeed config here I can also evaluate the output model with lm-evaluation-harness on single gpu with a non-one batch size. However, now I am using
model_training
to train a reward model. I can only run with the below setting on 8xA100(80G):with deepspeed config
zero3_config_sft.config
.(As you can see, it is very alike with the one above) In addition, I can not eval the output model usingeval_rm.py
on single gpu(even with batch size 1) because of GPU OOM. I didnt find any code that reduce GPU memory indolly
orlm-evaluation-harness
. And the modelGPTNeoXforCasualLm
should consume more memory thanGPTNeoXRewardModel
as I see the code of the output layer.