search

williamyang1991 / DualStyleGAN

[CVPR 2022] Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer
Other
1.61k stars 249 forks source link

the gpu memory usage of finetuning dualstylegan on 8 gpus #88

Open 1787648106 opened 1 year ago

1787648106 commented 1 year ago

Hi, thx for your excellent work! When I finetune the model on a 8 gpus per node device , I found that the gpu memory usage of the first card is about 2 times higher than that of the other seven cards. Is this normal?

williamyang1991 commented 1 year ago

This is because some of the pre-trained model is saved on GPU rather than CPU. So when you load these models before running the multi-GPU process, all models are loaded into GPU0, making GPU0 has high memory usage. You can manually check each of your loaded pre-trained models, and make sure they are saved and loaded into CPU before running the multi-GPU process.

1787648106 commented 1 year ago

Thanks for your reply, I'll try it later.