GAN Training script does not close everything properly

[whyhardt]

Running gan training several times in a row seems to deplete some resources Exact steps were:

1. Running DDP Training on Oscar for a few epochs
2. KeyboardInterrupt
3. Re-running training
4. Estimated time became twice as high

[whyhardt]

Could not reproduce on single GPU (non-DDP training) Two approaches; Check each individually:

1. added a try/except clause at DDP Training level to destroy the process group if an exception occurs
2. added a torch.cuda.empty_cache() call before each training run to clear the cache

[chadcwilliams]

In my most recent trainings, I also didn't see this issue emerge. I will try to reproduce it explicitly with GPUs on Oscar and on Google Colab. If I can't reproduce it, I'll close the issue.

[chadcwilliams]

Seems fixed now (could not reproduce the issue).