Validation/Testing takes a lot of time (More than 10 hours)

marcbelmont / cnn-watermark-removal

Fully convolutional deep neural network to remove transparent overlays from images

1.23k stars 227 forks source link

Validation/Testing takes a lot of time (More than 10 hours) #28

Closed kmrabhay closed 5 years ago

kmrabhay commented 5 years ago

I trained on dataset of 192 images. Train size = 192*0.8/2 = 153 and training on 39 images. I started training on google cloud with 24 GB Ram . 100 epoch completed with in 15 min but while evaluating after 100 epoch .. It is taking a lot of time. Its been 9 hours since evaluation part started after 100 epoch and its still not completed.

What can be done to speed it up? Thanks.

kmrabhay commented 5 years ago

Fixed it by removing while True: part in evaluation part(Line after saving the model) which was causing the evaluation process to go infinite loop resulting training never complete beyond one checkpoint after saving it.