ML kit could not recognize handwritten text.

[Chandra-Sekhar-Bala]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs in the code in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• For help troubleshooting your application that does not fall under one of the above categories, reach out to the personalized Firebase support channel.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: 2021.3.1
• Firebase Component: ML Kit
• Component version: 18.0.1

[REQUIRED] Step 3: Describe the problem

The ML kit can recognize digital photos very well, but in case of Handwritten text it fails badly.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[argzdev]

Thanks for reporting, @Chandra-Sekhar-Bala. Can you provide more details of how it fails badly? Are there any specific setups, language, writing, etc.? Also any chance you could provide a MCVE? It'll greatly help us alot. Thanks

[Chandra-Sekhar-Bala]

The issue I'm facing : Demo:

https://user-images.githubusercontent.com/57448981/192111321-d3acd164-8356-49ac-b156-d58d891f3ba4.mp4

image:

I've used Firebase ML kit, and you can see that it cannot recognize the handwritten text; This demo is from this application

[argzdev]

Hi @Chandra-Sekhar-Bala, thanks for the extra details. From the looks of it, the SDK is working as intended and is being able to recognize some "text". However, as you can tell, there are limitations on what it can decipher. The letters has to be legible enough for the SDK to understand the given text. In this case, I don't think the SDK was built to fully recognize any sorts of written handwriting. Here are some use cases provided in the documentation:

Text recognition can automate tedious data entry for credit cards, receipts, and business cards. Cloud-based API, you can also extract text from pictures of documents, which you can use to increase accessibility or translate documents. Apps can even keep track of real-world objects, such as by reading the numbers on trains.

What we can do for now is mark this as a feature request, so that our engineers can take a look into it once they have the time. Feel free to add any additional information you may find that you think will help. And while I’m unable to promise any timeline for this, if others find this useful adding an emoji thumbs up on the original post can help our engineers prioritize adding this to the roadmap. Thanks!

[thatfiredev]

@Chandra-Sekhar-Bala You can also try out ML Kit Digital ink recognition. But there are 2 important things I should point out:

1. It requires handwritten text to be inputted directly in the app (eg. using a Canvas), which means it's not suitable for reading handwritten text from an image (like you did on your sample video).
2. Digital ink recognition is not a Firebase product - it's maintained by the ML Kit team (https://developers.google.com/ml-kit/).

[Chandra-Sekhar-Bala]

@argzdev Thanks for the information and adding feature request tag. I look forward to see this feature live :)

[Chandra-Sekhar-Bala]

@Chandra-Sekhar-Bala You can also try out ML Kit Digital ink recognition. But there are 2 important things I should point out:

1. It requires handwritten text to be inputted directly in the app (eg. using a Canvas), which means it's not suitable for reading handwritten text from an image (like you did on your sample video).
2. Digital ink recognition is not a Firebase product - it's maintained by the ML Kit team (https://developers.google.com/ml-kit/).

Thanks for that :D