search

awslabs / handwritten-text-recognition-for-apache-mxnet

This repository lets you train neural networks models for performing end-to-end full-page handwriting recognition using the Apache MXNet deep learning frameworks on the IAM Dataset.
Apache License 2.0
488 stars 189 forks source link

How to compute labelling probability after prediction? #10

Closed shrinidt closed 4 years ago

shrinidt commented 5 years ago

Hi,

Thanks a lot for this wonderful piece of work.

I am trying to calculate CTC loss to compute labeling probability after prediction. Please guide if it is possible

Thanks

jonomon commented 5 years ago

Could you please elaborate on what you are trying to do?

shrinidt commented 5 years ago

Hi, I actually want to have a score which tells me with what confidence can i claim that my prediction for a handwritten text is correct?.

jonomon commented 5 years ago

Hi shrinidt, A handwriting recogniser can be evaluated with the Word Error Rate (WER) or Character Error Rate (CER).

In our implementation, we used SCLITE to calculate it. You can see it here.

I hope this answers your question.

shrinidt commented 5 years ago

Hi jonomon,

Thanks for the reply. I understand what you mean. But my problem is this. I do not have ground truth(actual text) with me. So whenever I predict a handwritten text,can i log a confidence score along with it, telling how confident I am about the prediction. Something like this-

No 3 in https://towardsdatascience.com/faq-build-a-handwritten-text-recognition-system-using-tensorflow-27648fb18519

Thanks

ThomasDelteil commented 5 years ago

Because of how the CTC collapsing work, any confidence score would be quite hard to compute and rely on, because of repetitions, multiple possible paths, etc. However i you want a score that might be helpful to you, you could softmax normalize the prediction of each time step, and use the average probability for the characters of the final path as an indication of confidence.

For example, with [a,b,c,{}] as a dictionary, sequence length of 6.

a  0.1 0.0 1.0 0.1 0.0 0.0
b  0.2 0.1 0.0 0.7 1.0 0.0
c  0.7 0.8 0.0 0.1 0.0 0.0
{} 0.0 0.1 0.0 0.1 0.0 0.9

"ccabb{}" => cab" after CTC collapsing

(0.7+0.8+1.0+0.7+1.0+0.9) / 6 = 0.85

The average confidence score is 85%

shrinidt commented 4 years ago

Thanks Thomas. I tried this with our model maximum sequence length being 100. I seem to get maximum score as 0.85 and nothing beyond. Am I doing something wrong?

ThomasDelteil commented 4 years ago

no it seems like a reasonable score