Traing issue:“torch.autograd.set_detect_anomaly(True)”

icon-lab / SynDiff

Official PyTorch implementation of SynDiff described in the paper (https://arxiv.org/abs/2207.08208).

Other

244 stars 37 forks source link

Traing issue:“torch.autograd.set_detect_anomaly(True)” #43

Open Luke-Huang-Github opened 6 months ago

Luke-Huang-Github commented 6 months ago

Environment： GPU： single RTX4090, Dataset: T1 <-> T2 in IXI dataset Number of images: 2674 pairs In "train.py" file, when I deleted this line of code：“torch.autograd.set_detect_anomaly(True)” , the speed of training one eopch is faster , but the result of psnr and ssim get worse sometimes. Time consumption in one epoch: [original: about 1.5hour ; deleted the code: about 36 minutes] PSNR: [original: 29.5 ; deleted the code: 25.5]

Could you please tell me how to solve this problem?

icecreamlovepudding commented 3 weeks ago

I meet the same question ,the training speed is so slow. Training 500 pairs of images, need 7days for only 50epoch,do you know how to solve this question? Thank you !