I have been attempting to train the StarGAN v2 model using an RTX 3090 GPU, but I encounter a persistent issue where the loss values turn to NaN, and the generated images are solely composed of noise. This is puzzling, especially since the same code and data work flawlessly on a Tesla P100 GPU.
Could you please shed some light on what might be causing these discrepancies between the two GPUs? Any insights or suggestions you could provide would be immensely appreciated, as I am quite perplexed by this behavior.
I have been attempting to train the StarGAN v2 model using an RTX 3090 GPU, but I encounter a persistent issue where the loss values turn to NaN, and the generated images are solely composed of noise. This is puzzling, especially since the same code and data work flawlessly on a Tesla P100 GPU.
Could you please shed some light on what might be causing these discrepancies between the two GPUs? Any insights or suggestions you could provide would be immensely appreciated, as I am quite perplexed by this behavior.