search

Belval / CRNN

A TensorFlow implementation of https://github.com/bgshih/crnn
MIT License
299 stars 101 forks source link

Training time #40

Open sharonapa opened 5 years ago

sharonapa commented 5 years ago

Hi, can you approximate train time of 100k examples on Gtx 1080? I started it, seems very slow. Thanks.

Belval commented 5 years ago

The current pretrained model (used to make the predictions in the README) took about 24-48 hours to train on a GTX1080Ti. You'd be looking at 72 hours-ish for a GTX1080.

If you feel adventurous, I am pretty sure that there are a lot of low-hanging fruits in the training code that could speed up the whole thing. Current data loading is not optimal for example.

sharonapa commented 5 years ago

Thanks, I've noticed it's taking some time to process one batch. I've few more questions : 1) where can I see your 200k data set? What was the parameters you ran the training? 2) the loss value I saw started from 6900 and dropping. Is this normal value? 3) my parameters were: 100k data set, batch size 32, 100k iterations (too much?) 4) can I speed up the lstm cells(or other ways to speed up) ? I've read that cudnnlstm does support variants sequence length. Thanks again.

בתאריך יום א׳, 31 במרץ 2019, 16:05, מאת Edouard Belval ‏< notifications@github.com>:

The current pretrained model (used to make the predictions in the README) took about 24-48 hours to train on a GTX1080Ti. You'd be looking at 72 hours-ish for a GTX1080.

If you feel adventurous, I am pretty sure that there are a lot of low-hanging fruits in the training code that could speed up the whole thing. Current data loading is not optimal for example.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Belval/CRNN/issues/40#issuecomment-478340057, or mute the thread https://github.com/notifications/unsubscribe-auth/AKS0brt1zJF7si7Rt5O_cKNLvKRXuWKAks5vcLKmgaJpZM4cULKs .