search

jeya-maria-jose / KiU-Net-pytorch

Official Pytorch Code of KiU-Net for Image/3D Segmentation - MICCAI 2020 (Oral), IEEE TMI
https://sites.google.com/view/kiunet/kiu-net
MIT License
357 stars 80 forks source link

How much time cost for one epoch when you trained the KiUNet? #17

Closed Gitjasonjin closed 3 years ago

Gitjasonjin commented 3 years ago

Good work and thanks for your shared code. When I train the network with my own datasets, it seems really slow, about 16 mins for one epoch with Tesla V100. Which operation takes the most time?

jeya-maria-jose commented 3 years ago

Thank you for the interest in our work. For one epoch (batch size 1, image resolution: 128*128) , the training time is 32 seconds while using a RTX 2080 Ti GPU. One thing to be noted is that the first epoch takes more time as we run inference on validation set in the first epoch (takes around 90 seconds). So , if you are validating a lot of images at each epoch it would take a lot of time. I would suggest you to make the --save_freq high so that more time is not lost in validation. If you use high resolution input, try using patch wise training to save training time.

jeya-maria-jose commented 3 years ago

Hi @Gitjasonjin , so there was a bug in the train file (the model was overwritten on each iteration) which might have increased the training time. The process should be faster now.