Failing to detect a QR code in a certain angle

[nyyManni]

I am utilizing libquirc for decoding QR codes from a smartphone's screen with a camera. The library has been working really well so far. However, I just discovered that there is a case where having the phone in a very specific orientation causes the library to find the QR code corner markers in invalid locations, and then fails to read the data with an ECC failure. An example image where the decoding fails is shown below:

The interesting part is, that if I rotate the image 90 degrees clockwise, the QR code is detected correctly. Also, if I photoshop away the text that is visible in the phone's display, the detection is successful.

Any ideas on how to fix or work around this?

[dlbeer]

On Mon, Dec 10, 2018 at 04:51:38AM -0800, Henrik Nyman wrote:

I am utilizing libquirc for decoding QR codes from a smartphone's screen with a camera. The library has been working really well so far. However, I just discovered that there is a case where having the phone in a very specific orientation causes the library to find the QR code corner markers in invalid locations, and then fails to read the data with an ECC failure. An example image where the decoding fails is shown below:

The interesting part is, that if I rotate the image 90 degrees clockwise, the QR code is detected correctly. Also, if I photoshop away the text that is visible in the phone's display, the detection is successful.

Any ideas on how to fix or work around this?

I don't know of an easy fix right away, but inspect shows that it mistakenly identified a chunk of the grid as a finder pattern and decided that it aligned well with the other two correctly-identified patterns.

Probably the best route to go towards fixing it would be to see if you can get the finder scan to be stricter about what constitutes a valid capstone. This is easier if you know that the angles you're going to have to deal with in practice won't be too difficult. For example, if you modify the err assignment in finder_scan() to:

err = avg * 1 / 4;

...then the image you're having trouble with is decoded fine.

Cheers, Daniel

-- Daniel Beer dlbeer@gmail.com http://dlbeer.co.nz/ PGP: BA6E 0B26 1F89 246C E3F3 C910 1E58 C43A 160A 553B

[nyyManni]

Thanks, that indeed does solve the sample case.

I did a more thorough test by rotating the image with OpenCV with every angle between 0 and 360, with steps of 0.2 degrees. Almost all of the images were decoded successfully, except a few which were rotated ~10 degrees from the original picture that I shared with you. Here is one failing one. For me it says "Format data ECC failure":

By performing a normalization (cv2.equalizeHist) to the image, that one is also successfully decoded. Modifying the err assignment in finder_scan does not seem to have any effect with this one.

Additionally, since our use case is quite specific, do you have any suggestions on how we could customize the library to be more robust with these kind of images (white screen, QR code in the center covering 90% of its width, and an otherwise dark environment)? What functions should we focus on tuning?

[stuta]

I have this picture from scanner. It does not recognize it in this normal orientation or upside-down, but it does recognize it when rotated 90 degrees to left or to right.

Is there any way to fix this? Could we have 'try-harder' option when it rotates the picture if nothing was found and tries again with rotated version?

[dlbeer]

My guess is that the thin horizontal white stripe pattern on the image is what's causing a problem. Have you tried applying a filter to remove that? A column-oriented hysteresis scan or even a simple dilation filter might be enough.

[stuta]

Like I said it works great with rotated image. I guess that rotating image is much faster than applying filters. But memory allocation is very slow in computers and creating a new image is slow and memory consuming.

By far the fastest would be not to change the image, but change detection algorithm by reading same image 90 degrees rotated - just a coordination change. And because most printers will produce vertical lines (like in my example) it might be best to start reading image from left to right - or right to left.

Option for image reading direction would be great. Then we could call a new read if one direction fails.

[dlbeer]

stuta notifications@github.com writes:

Like I said it works great with rotated image. I guess that rotating image is much faster than applying filters. But memory allocation is very slow in computers and creating a new image is slow and memory consuming.

That's a bit surprising -- the kind of filter I was thinking of would be pretty much linear access with minimal processing (just check neighbouring pixels and apply a simple bitwise operation). I would expect it to be at least as fast as rotation by 90 degrees. What were you trying?

You could either apply the filter or rotate the image in-place. I don't see any need for additional memory allocation (but if you want to, you could preallocate a buffer and reuse it).

Cheers, Daniel

-- Daniel Beer dlbeer@gmail.com http://dlbeer.co.nz/ PGP: BA6E 0B26 1F89 246C E3F3 C910 1E58 C43A 160A 553B

[stuta]

I did not find a code to rotate image in-place without memory allocations (and I think it is impossible). This was simplest I found: https://codereview.stackexchange.com/questions/29618/image-flip-algorithm-in-c.

Images I use are scanned 600 px/inch so their size is huge. Changing the algorithm to scan from another 90° angle is the fastest way (but harder to do) because no memory allocations are needed. Others have had problems with angle, I think this would solve their problems too.

Easy option would be to add that simple flip algorithm and give option "try-flipped". I'm using load_png and load_jpeg and that's why this flipping should be done inside quirc library.

[dlbeer]

stuta notifications@github.com writes:

I did not find a code to rotate image in-place without memory allocations (and I think it is impossible). This was simplest I found:

https://codereview.stackexchange.com/questions/29618/image-flip-algorithm-in-c.

You can do it easily for a square image. But you don't need to allocate for each frame anyway, because you can allocate one buffer and reuse it.

Images I use are scanned 600 px/inch so their size is huge. Changing the algorithm to scan from another 90° angle is the fastest way (but harder to do) because no memory allocations are needed. Others have had problems with angle, I think this would solve their problems too.

I think this is unlikely to be fast, because doing a vertical finder scan will skip across cache lines. There are cache-friendly 90-degree rotation algorithms which I'd expect to be much faster.

Easy option would be to add that simple flip algorithm and give option "try-flipped". I'm using load_png and load_jpeg and that's why this flipping should be done inside quirc library.

Or, you could pre-process your image to remove the artifacts you have. I'm not talking about something slow like a convolution filter -- just a simple scan-line oriented hysteresis step.

Have you tried either of these solutions?

Cheers, Daniel

-- Daniel Beer dlbeer@gmail.com http://dlbeer.co.nz/ PGP: BA6E 0B26 1F89 246C E3F3 C910 1E58 C43A 160A 553B

[stuta]

I would need another library for reading images, removing the artifacts and convert images to format that qirc uses. Also in that case the memory requirements double because of creating a new altered image. Memory consumption is already huge.

[dlbeer]

stuta notifications@github.com writes:

I would need another library for reading images, removing the artifacts and convert images to format that qirc uses. Also in that case the memory requirements double because of creating a new altered image. Memory consumption is already huge.

No, you're confusing two different things. You would need to allocate a second buffer in order to rotate a non-square image, which is one of your possible solutions -- but never mind that (although I notice now that your example image is square).

To remove the artifacts you don't need to allocate anything, you don't need any libraries or format conversions, and you can do it in-place in quirc's existing buffer in about 20 or so lines of code.

I would suggest trying a median filter: for each pixel in the image buffer, collect it, plus the two pixels above and below into a 3-element array. Sort the array using a 3-step sorting network, then select the middle element to put back.

I'm not inclined to put a filter of this kind into the library, as it solves a very application-specific problem, and it's already easy to implement in application code without any modifications to the library.

Cheers, Daniel

-- Daniel Beer dlbeer@gmail.com http://dlbeer.co.nz/ PGP: BA6E 0B26 1F89 246C E3F3 C910 1E58 C43A 160A 553B

[yangh-zzf-itcast]

No, you're confusing two different things. You would need to allocate a second buffer in order to rotate a non-square image, which is one of your possible solutions -- but never mind that (although I notice now that your example image is square). To remove the artifacts you don't need to allocate anything, you don't need any libraries or format conversions, and you can do it in-place in quirc's existing buffer in about 20 or so lines of code. I would suggest trying a median filter: for each pixel in the image buffer, collect it, plus the two pixels above and below into a 3-element array. Sort the array using a 3-step sorting network, then select the middle element to put back. I'm not inclined to put a filter of this kind into the library, as it solves a very application-specific problem, and it's already easy to implement in application code without any modifications to the library. Cheers, Daniel

Hello，When I tested the library, capstones could detect the rotation, but num_ Grid=0, what's the problem？Thanks very much

[dlbeer]

Hello，When I tested the library, capstones could detect the rotation, but num_ Grid=0, what's the problem？Thanks very much

There are a lot of things that might prevent a code from being recognized. It might help to try loading your image with the inspect tool.