Again, thank you for making such a great tool available. I am writing because I am having some issues with loading from a checkpoint model. The training runs totally fine initially, but when I want to resume from a model checkpoint (I am resuming at the 20,000th step), it cannot go past 1 step without running out of gpu memory. I am running the training on 6 V100's and here is my command:
`python run_train.py --gan_type StyleGAN2 --gan_resolution 256 --w_shift True --gan_weights /XXX/work/stylegan2-pytorch/500kNFT_tf_stylegan2.pt --deformator ortho --out /XXX/work/Notebooks/GAN/directionality/out/NFT_500k_ortho --multi_gpu 6 --shift_predictor_size 256`
Hello,
Again, thank you for making such a great tool available. I am writing because I am having some issues with loading from a checkpoint model. The training runs totally fine initially, but when I want to resume from a model checkpoint (I am resuming at the 20,000th step), it cannot go past 1 step without running out of gpu memory. I am running the training on 6 V100's and here is my command: `python run_train.py --gan_type StyleGAN2 --gan_resolution 256 --w_shift True --gan_weights /XXX/work/stylegan2-pytorch/500kNFT_tf_stylegan2.pt --deformator ortho --out /XXX/work/Notebooks/GAN/directionality/out/NFT_500k_ortho --multi_gpu 6 --shift_predictor_size 256`