Training nodule detector is slow. 16 hours for an epoch

juliandewit / kaggle_ndsb2017

Kaggle datascience bowl 2017

MIT License

624 stars 290 forks source link

Training nodule detector is slow. 16 hours for an epoch #16

Closed jkrudra closed 7 years ago

jkrudra commented 7 years ago

Hi,

I'm running step2_train_nodule_detector.py in Linux machine with TitanX GPU. It's taking close to 16 hours for completing a single epoch, where as in Readme.MD it's mentioned the total time for 12 epochs is 8 hours. I'm using anaconda2 python environment.

Can you please help me with this ? What am I missing ?

jkrudra commented 7 years ago

Hi,

Found the issue. This is because of the steps_per_epoch value given in line number 387(fit_generator function). I changed the value from len(train_files) / 1 to len(train_files) / batch_size. Now it's taking approximately an hour for an epoch.

This issue can be closed.