search

cozmo / jsQR

A pure javascript QR code reading library. This library takes in raw images and will locate, extract and parse any QR code found within.
https://cozmo.github.io/jsQR/
Apache License 2.0
3.64k stars 602 forks source link

How to guide visually impaired user to bring QR code into view #164

Closed iowagrade closed 3 years ago

iowagrade commented 4 years ago

I am working on an app for visually impaired people. I want to be able to recognize if a partial QR code is potentially in the view and help guide the user to bring the full QR code into the view.
ie. For example, I would want to recognize if a partial potential QR code is found and if it was, say, found on the right side of the view, guide the user to move the camera to the right to bring the rest of the QR code into view for decoding.

I am going through the code now, but thought I would throw it out there to see if anybody has had this experience or enough experience with the theory of the code to guide me / give me a kick in the right direction. Thanks!

cozmo commented 4 years ago

I don't have any great ideas here unfortunately. The way the library is currently written the locator (which tries to detect things that look like QR codes) is pretty noisy - It finds a ton of things that look like tracking patterns (the 3 squares in the corners of a QR code) and then tries to find 3 that look similar enough to each other/are aligned in such a way that we can develop confidence they're actually a QR code. As such, before we find 3 tracking patterns there's not much information that could be used to guide the user, and after we find the 3 tracking patterns, we're probably able to scan the QR.

You could potentially do something where you develop a sort of heat map of the image, based on things that look like they could be QR codes (low color variance, lots of straight lines etc), and then use that to guide the user.

For what it's worth, this is something that machine learning it pretty well suited for. In fact, we've had great success using ML to replace the locator step, but the generated models are too large to efficiently use on the frontend.

Sorry I can't be more helpful, and happy to answer other questions you have, but the current architecture of jsQR makes this fairly difficult I think.

iowagrade commented 4 years ago

Thanks for the reply Cozmo. I have been going down the ML path on the problem using the openCV library. I am in the process of gathering images with qr codes in them to do the haar training.

As I am not overly familiar with using the openCV library, one of the questions I am contemplating is whether a model trained with full qr codes in view would be able to recognize partial qr codes or not, both in the respect of a qr code partially in the image from being at the image edge or from being on something like a can that is fully in the image but turned in a direction such that only a partial qr code is showing. I would enjoy hearing your thoughts on this if you have any, otherwise, again, thanks for your reply and great work on the library!