search

zaccharieramzi / fastmri-reproducible-benchmark

Try several methods for MRI reconstruction on the fastmri dataset. Home to the XPDNet, runner-up of the 2020 fastMRI challenge.
https://fastmri.org/leaderboards
MIT License
151 stars 50 forks source link

Time per epoch increasing #184

Open SivaHitesh047 opened 1 year ago

SivaHitesh047 commented 1 year ago

Hey, While running the code (nc_train) on the GPU, the time per epoch keeps increasing after each epoch, making it difficult to conduct experiments. The First epoch starts with 30 mins ETA and by the time it gets to 4th or 5th epoch it's 4 hrs. Do you know the reason why this is happening? If yes, please suggest anything to fix this.

Thanks and Regards, Siva Hitesh

zaccharieramzi commented 1 year ago

Hi @SivaHitesh047 ,

Sorry for getting back to you so late, I was on holiday for a while. I have not seen this behaviour before. Can you copy-paste here exactly what you run in order to see the epoch time increase?

Cheers