search

Belval / CRNN

A TensorFlow implementation of https://github.com/bgshih/crnn
MIT License
297 stars 101 forks source link

The loss always be inf #33

Open Dinghow opened 5 years ago

Dinghow commented 5 years ago

I trained with the data generated by your tool TextRecognitionDataGenerator. And trained with 100 iterations, the loss is always be inf, I'm wondering the reason, thx a lot.

Belval commented 5 years ago

Hi, inf loss does happen from time to time but usually, it's either a random occurrence (so restarting the training will fix it) or a problem with the dataset.

How big is your dataset? Are you training for Chinese? If so, what is the length of your CHAR_VECTOR?

Thank you

Dinghow commented 5 years ago

Thanks for your immediate reply. Tried to restart the training many times, without any improvement. BTW, the size of my training dataset is 8000 in english. I just added some punctuations like ,.;'" to the CHAR_VECTOR.

Belval commented 5 years ago

Interesting, I'll do some testing from my side see if I can get a test case.

If you could compress your 8000 samples dataset and send it to me to ensure reproducibility I would be grateful.

You can drop the file here: https://nextcloud.belval.org/index.php/s/MQsQrK8egatQtLo

Dinghow commented 5 years ago

Sure, I'm glad to do that, the size is 10001 actually. And my CHAR_VECTOR is CHAR_VECTOR = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-'.!?,\"" just reuse yours.

Belval commented 5 years ago

Just a heads up to say that I was able to reproduce the issue. I'm quite busy at the moment but I'll try to troubleshoot it ASAP.

Thank you for bringing this to my attention.

Dinghow commented 5 years ago

OK, thx for your attention, hope for your update.

daming98 commented 5 years ago

I face the same problem. Have you solved it?

daming98 commented 5 years ago

My dataset is Chinese and the length of CHAR_VECTOR is 70000+, the long lengths can cause problems?

daming98 commented 5 years ago

My dataset is Chinese and the length of CHAR_VECTOR is 70000+, the long lengths can cause problems?

sorry, my CHAR_VECTOR is 7000+.

The training process is too slow. How can I solve it? It trains five steps in four days.

Dinghow commented 5 years ago

I face the same problem. Have you solved it?

Not yet. I used this project without this problem https://github.com/wushilian/STN_CNN_LSTM_CTC_TensorFlow/blob/master/utils.py

daming98 commented 5 years ago

I face the same problem. Have you solved it?

Not yet. I used this project without this problem https://github.com/wushilian/STN_CNN_LSTM_CTC_TensorFlow/blob/master/utils.py

Thank you. I'll try.