Open 1787648106 opened 1 year ago
This is because some of the pre-trained model is saved on GPU rather than CPU. So when you load these models before running the multi-GPU process, all models are loaded into GPU0, making GPU0 has high memory usage. You can manually check each of your loaded pre-trained models, and make sure they are saved and loaded into CPU before running the multi-GPU process.
Thanks for your reply, I'll try it later.
Hi, thx for your excellent work! When I finetune the model on a 8 gpus per node device , I found that the gpu memory usage of the first card is about 2 times higher than that of the other seven cards. Is this normal?