search

ajbrock / BigGAN-PyTorch

The author's officially unofficial PyTorch BigGAN implementation.
MIT License
2.86k stars 475 forks source link

Out of memory #84

Open diaodeyi opened 3 years ago

diaodeyi commented 3 years ago

Why the program can run in the training, but stuck in saving weight?

Saving weights to /data1/code/BigGAN-PyTorch/weights/BigGAN_I512_seed0_Gch96_Dch96_bs8_nDa8_nGa8_Glr1.0e-04_Dlr4.0e-04_Gnlinplace_relu_Dnlinplace_relu_Ginitortho_Dinitortho_Gattn64_Dattn64_Gshared_hier_ema... Saving weights to /data1/code/BigGAN-PyTorch/weights/BigGAN_I512_seed0_Gch96_Dch96_bs8_nDa8_nGa8_Glr1.0e-04_Dlr4.0e-04_Gnlinplace_relu_Dnlinplace_relu_Ginitortho_Dinitortho_Gattn64_Dattn64_Gshared_hier_ema/copy0...

RuntimeError: CUDA out of memory. Tried to allocate 3.75 GiB (GPU 0; 11.91 GiB total capacity; 6.14 GiB already allocated; 416.75 MiB free; 10.94 GiB reserved in total by PyTorch)

diaodeyi commented 3 years ago

even though the Occupation rate before is only 50%, after the step 1000 ,it still out of memory. I change the code in the utils, Device('cuda') -> Device('cpu') it seems nothing

Wanghd-MVP commented 3 years ago

I have the same problem，do you have solved it？ If you have solved it, can you tell me how to solve this problem. Thank you very much!!!!!!!