Broad research on seven-segment display libs

[vincent-karuri]

Look into these links (and perhaps others) for clues on what's out there for 7-segment-display capture systems:

1. https://github.com/SachaIZADI/Seven-Segment-OCR
2. https://github.com/alexcohn/tess-two
3. https://www.youtube.com/watch?v=J0jg1Hj2GhU
4. https://code.google.com/archive/p/serak-tesseract-trainer/downloads
5. https://stackoverflow.com/questions/17672705/text-detection-on-seven-segment-display-via-tesseract-ocr
6. https://play.google.com/store/apps/details?id=jp.gr.java_conf.coskx.ddreader2.trial&hl=en_US&gl=US
7. https://github.com/arundil/OCR7Segments/tree/master/OCR7SegmentsApp
8. https://github.com/arturaugusto/display_ocr (recommended by UW)
9. https://github.com/jiweibo/SSOCR/blob/master/ssocr.py (a single script)

[vincent-karuri]

@shoaibmushtaq25 could you indicate which of these you'll be picking up?

[shoaibmushtaq25]

Currently, I am working on this one (number 7) https://github.com/arundil/OCR7Segments/tree/master/OCR7SegmentsApp Converted old project to androidx and now trying to run it on mobile.

[vincent-karuri]

MLKit work is documented here https://github.com/OpenSRP/fhircore/issues/69.

[shoaibmushtaq25]

After that, I will be testing number 6 - https://play.google.com/store/apps/details?id=jp.gr.java_conf.coskx.ddreader2.trial&hl=en_US&gl=US

[vincent-karuri]

I'll try 1

[f-odhiambo]

Here is a link to all the images to related lighting conditions while reading the G6PD device https://drive.google.com/drive/folders/1TcdC46RpUBtaAygfDclFSUrLon2QYI8m?usp=sharing

[vincent-karuri]

thanks @f-odhiambo something to note here is at some point we need to label the images with the actual readings on them if we go the route of training our own model. And probably generate our own?

[shoaibmushtaq25]

Update so far, 7- I have converted the old eclipse project to androidx and run it on android studio and finally able to install the app in the mobile. But when I go to run it, it needs to install openCV package and looks like it's taking ages to install this openCV package and it's installing since hours. Let see what happens after that.

6- This was a paid app on playstore. I purchased it and tried scanning our Standard G6PD device images with this app. So this app is able to detect the Seven segment but it can scan only one word at a time. Also I am going to dig down and search to check if we can get the source code of this app available in our usable form.

Here are the resulted images (attaching few here as well) after this scanning. https://drive.google.com/drive/folders/1k8PHXwSljwPR-iQf-VBY6Aq1FDJAuAY4?usp=sharing

[shoaibmushtaq25]

So far, no luck for source code or open source library of Yomzo (number 6)

[vincent-karuri]

@shoaibmushtaq25 is there information on how 6 can be launched from another app and to get its results after capture?

[vincent-karuri]

• 6 seems to perform well on the readings https://github.com/OpenSRP/fhircore/issues/82#issuecomment-821078171. It's however not opensource and there's no clear information on how we can launch it from another app and get the extracted readings. Next steps here would be:

  • Find out if we can get a library (APIs) from the developer (preferred)

  • Alternatively, find out if we can get information about launching the app and getting a reading from it

  • 8 has a lot of issues building (is outdated and doesn't have a requirements.txt). I could not get it to successfully run and so couldn't evaluate it. Next steps here would be:

  • Try see if we can contact the developers for some guidance

  • Dedicate time to try and upgrade the outdated parts, test and evaluate

  • 1 takes quite a while to train (been running for over 3 hrs) but for their dataset, it seems that each epoch provides better results. Next steps here would be:

    • train the model
    • crop out G6PD readings and add them to a test folder to see how it performs
    • If above is good OK, otherwise we'd need to train it with labelled images and see if the results are better
    • It also doesn't seem to be trained on capturing the entire floating point number (from the README images)

  • I haven't gone through 2 and 4 but it looks like we'd need to know how to work with Tessaract to train a model with the data? In that case, next steps are:

    • We'd need to invest time to understand how this works and then evaluate performance.

  • 9 has an interesting approach to encoding the 7-segment display as bits and is a single script which looks simpler. It however only works for that particular display but from skimming it looks as though it's because you have to tell it where to look for the digits (might be a small tweak but not sure). Next steps:

    • see if we can tweak where it looks for digits and evaluate performance

In most if not all of these, we want to be able to successfully run and subsequently evaluate whether the out-of-box solution works or not. I'm not sure we're a position to know exactly what to tweak right now or after the evaluation but at the least we can try out the solution as is.

[vincent-karuri]

results of running Model_Single() from 1 on their training set. Click on the images to enlarges (gives ~80% accuracy).

[pld]

Cool, let's spend Monday to finish looking into (7), which I think is based on (2).

For (2) yes we'd either need to train of find a model, there should be a model for (2) in https://github.com/arundil/OCR7Segments (ie 7)

[shoaibmushtaq25]

6) For number 6, I tried to find out source code or open source libraries used in it but no luck and what I did this morning is that I sent out an email to the email-id mentioned on the Google Play store link of this app "Yomzo" but still there is no reply from them.

7) For number 7, I invested a reasonable time to convert the old eclipse project to android studio project and also converted it to androidx imports and was able to install this app successfully in the mobile. When I run this app, this asks for openCV manager package to install in the mobile. I tried many different ways (Direct from the app or separate installation through terminal) to install the OpenCV manager package and finally I was able to install it few times before. After installing OpenCV manager package, I run the app but the app is not detecting anything and only showing a rectangle/square of ROI(region of interest). The app also crashes off and on. here is the screenshot of number 7 app.

[pld]

Let's continue looking 7 for another day, don't worry about the crashing, let's see if we can get any of the text rec to work

[shoaibmushtaq25]

As I mentioned in the comment here that I sent out an email to developer of "Yomzo" numeral 7-Segment app to ask about the source code or open source library or any way to launch "Yomzo" app from external app. I just received a reply from them. See image below for the details..I will paste it in the github issue as well for the record.

[pld]

@shoaibmushtaq25 what was the result of additional research into 7?

[shoaibmushtaq25]

@pld I invested yesterday on number 7 and tried to make it work after resolving some outdated code but the app still not working correctly and also crashes off and on while scanning readings through camera and scan didn't give any results. I configured NDK and some other required things to make it work but unfortunately no luck.

I am also trying today to make it work in parallel to FHIR core/sdk assignments.

[pld]

Yes of course, K that's helpful thank you, do you see any value in spending more time working on this?

[shoaibmushtaq25]

@pld I spent time today as well but still there were some errors coming out due to which app is not working as expected. Actually, there is a native C and C++ code involved in the app as well and looks like I am not able to properly include that native code in the apk as i researched on the error coming out. I am not sure what would be the next steps now onwards.

[shoaibmushtaq25]

After hours of effort, finally I managed to add the native C/C++ code (.so files) in the final apk of number 7 and then tried to scan the G6PD device readings with it but unfortunately the app doesn't work as expected and crashes too frequently. Looks like the code is too old and not working as expected. I don't think to invest more time on it would be beneficial or give any good results. cc: @pld @f-odhiambo @vincent-karuri

[pld]

k thanks @shoaibmushtaq25 for your efforts, looks like we've definitely hit the end of road wrt options we listed here

[vincent-karuri]

https://github.com/arturaugusto/display_ocr @shoaibmushtaq25 for this, do you have the script with your changes made but which was failing?

[shoaibmushtaq25]

Yeah, I have its requirement.txt file which I get it prepared from my friend having expertise in python.

[vincent-karuri]

a few others that may be interesting:

10. https://github.com/kk5190/Reader-Of-Appliance-Display (Android)
11. https://github.com/NatholBMX/ScaleDigitizer looks easy to set up (python)
12. https://github.com/arturaugusto/display_ocr have not been able to run this (python)
13. https://www.unix-ag.uni-kl.de/~auerswal/ssocr/ (terminal)
14. https://github.com/auerswal/ssocr github code for above
15. https://github.com/FANMixco/7-segment-ocr-reader (C# inspired by above)
16. https://github.com/DevashishPrasad/LCD-OCR single script (python)
17. https://github.com/suyashkumar/seven-segment-ocr (python)
18. https://github.com/tvwerkhoven/heat_meter_reader looks specific to heat meters? (python)

[shoaibmushtaq25]

Here are the steps and requirements.txt file to run https://github.com/arturaugusto/display_ocr

1. Install Python 3.8 if not installed yet
2. Add Input Image path that you want to process on line 64 of file py3_raspberry_pi.py
3. Create a venev using following command python3 -m venv venv_ocr
4. The activate the venv using command source venv_ocr/bin/activate
5. Then install requirements.txt file given below using command pip install -r requirements.txt in directory where you will place this file
6. Then enter command python py3_raspberry_pi.py and see the results

requirements.txt

[shoaibmushtaq25]

I tried number 10 from the list above (https://github.com/kk5190/Reader-Of-Appliance-Display (Android)) And got some reasonable results out of it. I have attached videos of the results. Have a look at the drive link here to check the accuracy of results with this android project. cc: @pld @vincent-karuri @f-odhiambo

[vincent-karuri]

I tried number 10 from the list above (https://github.com/kk5190/Reader-Of-Appliance-Display (Android)) And got some reasonable results out of it. I have attached videos of the results. Have a look at the drive link here to check the accuracy of results with this android project. cc: @pld @vincent-karuri @f-odhiambo

is there a reading that's displayed somewhere, the docs say that it will voice out the reading once it gets it?

[vincent-karuri]

Also @shoaibmushtaq25 could you try it out on a mobile phone?

[shoaibmushtaq25]

Okay, I will give it a try on mobile phone as well.

[shoaibmushtaq25]

is there a reading that's displayed somewhere, the docs say that it will voice out the reading once it gets it?

No, the reading shows only in the ROI(Region of Interest) shown above the reading. Also there is no voice as i observed the app.

[shoaibmushtaq25]

Also @shoaibmushtaq25 could you try it out on a mobile phone?

I am done with the research. I mostly tried it on tablet yesterday and made the video from my mobile phone. and as @vincent-karuri said to test it out on mobile phone as well. so today, I also tried it out on my mobile phone and results are the same as on tablet. But still looks like this app is also crashing sometime specially on tapping on a Button given on the screen. I didn't dig into the code yet to see the reason of crashing anyways. By the way, looks like this can also help us understand to build our solution in the right direction at-least.

[shoaibmushtaq25]

so I just reproduced the crash on the number 10 while tapping on a button given on the screen and came to know that this is also throwing the same error java.lang.UnsatisfiedLinkError and its obvious reason is not including the native C/C++ code in the final apk. Let me try quickly if I can make it work in this way.

[shoaibmushtaq25]

Anyways, I manged to resolve the error around native C/C++ code to add into the apk.

I was running the app on my mobile phone (android version 11) and openCV package manager was not installing successfully. Then I tried it on my tablet (android version 9) and I manged to install openCV package manager in it successfully.

App was giving the same results as I shared in the videos attached here . This app is also crashing off and on. So just like number 7, this number 10 does need to invest some more time to resolve all the old code issues.

For number 10 to run on the mobile device, you also need to install OpenCV Package Manger 3.0.0. Here is the link to the respective OpenCV Package Manager.

[shoaibmushtaq25]

I also found a stable apk of number 7 somewhere on the internet and tried it on my device but their results accuracy was not so good. Here is the link of stable apk of number 7 found on the internet.

Also for number 7 to work on the mobile device, you also need to install openCv Package Manager 3.1.0. Here is the link to respective OpenCV Package Manager.

[pld]

Thanks for the updates, please add related code here for now, https://github.com/opensrp/diagnostics/tree/main/g6pd

[shoaibmushtaq25]

@pld , Related code of number 10 is uploaded to the above url https://github.com/opensrp/diagnostics/tree/main/g6pd

[f-odhiambo]

Research summary - ongoing

1. Reviewed APK/Code Paid version ($2) version of Yomzo app plus the documentation - No improvement

   • Optical Character Recognition of Seven Segment Display (Python based) - link
   • link
   • Seven segment libs - links
   • optical-character-recognition - link
   • tesseract-ocr link
   • Review Google AutoML framework link . Also contacted Fred, who referred us to Thye Yeow Bok, ML Kit expert to try to help us with the research

2. Compiled list of literature read

   • Automated method for detecting and reading seven-segment digits from images of blood glucose metres and blood pressure monitors - link - Fred has some leads already connected with him and will be sharing contacts soon
   • Real-Time Detection and Reading of LED/LCD Displays for Visually Impaired Persons link
   • Seven Segment Optical Character Recognition link
   • Utilizing Smartphone-Based Machine Learning in Medical Monitor Data Collection: Seven Segment Digit Recognition - link - closest research work to G6PD
   • How to recognize Seven Segment Displays content with ssocr (Seven Segment Optical Character Recognition) in Ubuntu 16.04 - link can be server-based
   • Text detection and recognition in raw image dataset of seven-segment digital energy meter display - link
   • Measurement Data Recognition from SevenSegment Indicator by Mobile Device - link This is good research on current problems we are facing

[shoaibmushtaq25]

Here is the link to more demo videos of results with

• Google ML Kit
• Number 7 (https://github.com/arundil/OCR7Segments/tree/master/OCR7SegmentsApp)
• Number 10 (https://github.com/kk5190/Reader-Of-Appliance-Display)

Demo videos of number 10 with voice/ textual feedback of the scanned readings are here

[shoaibmushtaq25]

Installed and run desktop version of Tesseract OCR and tested it with different Input Images , i.e, English Text, Numeric Text, G6PD device reading Image, Cropped out seven segment digits and results are documented here

[vincent-karuri]

SSOCR initial findings: https://docs.google.com/document/d/1RAPfoZhV5koVR-HUrftMoe-Zs7YcfPfqljc35HJfUJA/edit

[shoaibmushtaq25]

Tried already available seven segment display trained data sets with desktop based Tesseract OCR at different resolutions . Results are here