Feature Request: Better decoding of blurry QR codes

[cloudspeech]

The BoofCV QR Benchmark page shows that recognizing blurry QR codes is still a problem.

Unfortunately, for mobile QR-code scanning with smartphones, coping with blurriness is a must, since it helps decode earlier and in more situations. User studies have shown that robust decoding and decoding speed are crucial acceptance factors.

As a feeble attempt to support this feature request, here are some pointers to academic papers that might help with improving BoofCV QR-code detection in the presence of blur:

• A 2016 PhD thesis from ETH Zurich: Wearable barcode scanning — Advancements in visual code localization, motion blur compensation, and gesture control
• A 2011 PhD thesis from ETH Zurich: An Efficient Bar Code Recognition Engine for Enabling Mobile Services
• Fast Blur Removal for Wearable QR Code Scanners
• Blur-resistant joint 1D and 2D barcode localization for smartphones
• A Regularization Approach to Blind Deblurring and Denoising of QR Barcodes
• An exemplary Deep-Learning paper on deblurring +code

[lessthanoptimal]

thanks for the paper list! I'll check it out and see if there's anything that can be used. The main issue with blurred images is that they screw up the corners on the locator patterns. One reason BoofCV's scanner is so far is that it doesn't need to try that hard to decode the image since it nails the correct bits. I was looking into ways to improve it by using other features to estimate the bit locations.

[alexander-toschev]

Hello @lessthanoptimal ,

I want try out working on this issue, can you please point me which technique BoofCV use for QRCode reading?

[lessthanoptimal]

I've yet to write a paper on it and it does differ significantly from most others. The key step here is finding the locator patterns, i.e. the square with a square inside.

• BoofCV's approach is not typical and is closer to how ARToolkit works. A binary image is created, blobs are found, then it fits a polygon to those blobs. 4-sided polygons are checked to see if there's an inner blob. I've left out a lot of details that make it faster/more robust.
• The most common technique, which handles blurred images better, involves scanning the image along the rows and looking at the up down patterns. Similar to how a bar code reader works.

For the scanning approach see page 60 of the QR Code Specification. The main reason BoofCV doesn't employ that approach is that approach is not rotationally invariant and doesn't give you precise corners. Since BoofCV has very precise corners it can find all the data very fast, but that makes it more susceptible to damage/blur.

[alexander-toschev]

But what if try enhance source image before passing to the detector? I understand that it will add overhead for the preprocessing. I red about BLOB approach.

[lessthanoptimal]

You might be able to eek out a little bit of performance doing that, worth a shot. If you can manage to improve performance (even if it is expensive) by a significant at a minimum an example will be added to show that can be done. Not every use case requires real-time performance. A lot of people want something that just works even if it's slow.

[alexander-toschev]

Hello, I made class below for process a couple of filters, which actually inrease quality up to 30%. I got image from camera and pass in to 3 filters:

• Remove high light;
• Remove low ligjt;
• No filter.

And merge results. recognize = recognize.subimage(x1, y1,x2,y2); s.process(recognize); mergeResult(res, s.getDetections(), s.getFailures()); res.imageHeight = recognize.height; res.imageWidth = recognize.width; Tools.LogTime("RECO "+prefix+":", start, logger); if (CamerasSettings.Value.writeDebugFiles){ String name = String.format("%s-qr_%d-%s.%s",prefix, res.success.size() ,RandomStringUtils.randomAlphanumeric(8), "bmp"); File outputfile = new File(name); ImageIO.write(dbg.getSubimage(x1,y1,x2-x1,y2-y1), "bmp", outputfile); }

I will work to port opencv functions to boofcv. @lessthanoptimal can you point me if any available in BoofCV?

`public class ImageTransformer implements WebcamImageTransformer { private BufferedImageOp filter = new JHFlipFilter(JHFlipFilter.FLIP_90CW); public double[][] sharpen_kernel = {{-1,-1,-1}, {-1,9,-1}, {-1,-1,-1}}; public Mat sh; private static boolean loaded = false; public ImageTransformer() { if (!loaded) { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); loaded = true; } sh = new Mat(3, 3, CvType.CV_16SC1); }

@Override
public BufferedImage transform(BufferedImage image) {
    var mat = BufferedImage2Mat(image);
    mat = removeHighlights(mat);
    var bi = Mat2BufferedImage(mat);
    return bi;
}

public ArrayList<BufferedImage> process (BufferedImage image){
    var mat = BufferedImage2Mat(image);
    var matCopy = mat.clone();
    mat = removeHighlights(mat);
    var bi = Mat2BufferedImage(mat);
    matCopy = removeHighlights(matCopy);
    var bi2 = Mat2BufferedImage(matCopy);
    var res= new ArrayList<BufferedImage>();
    res.add(bi);
    res.add(bi2);
    return  res;

}

public  Mat removeHighlights(Mat pic) {
    try
    {
        //Mat image = Imgcodecs.imread(pic,0);
        Mat sh = new Mat(3, 3, CvType.CV_16SC1);
        double[][] sharpen_kernel = {{-1,-1,-1}, {-1,9,-1}, {-1,-1,-1}};
        for (int i = 0; i < 3; i++) {
            for (int j = 0; j < 3; j++) {
                sh.put(i, j, sharpen_kernel[i][j]);
            }
        }
        Mat image = pic;
        Mat sharpen = new Mat();
        Imgproc.filter2D(image,sharpen, -1,sh);
        Mat img_thresh = sharpen;
        Imgproc.threshold(sharpen,sharpen,0,125,Imgproc.THRESH_TOZERO);
        return sharpen;
    }
    catch (Exception ex){
        return null;
    }

}

public Mat sharp(Mat pic){
    double[][] sharpen_kernel = {{-1,-1,-1}, {-1,9,-1}, {-1,-1,-1}};
    Mat sh = new Mat(3,3, CvType.CV_16SC1);
    for(int i=0; i<3; i++){
        for(int j=0; j<3; j++){
            sh.put(i,j, sharpen_kernel[i][j]);
        }
    }
    Mat sharpen = new Mat();
    Imgproc.filter2D(pic,sharpen, -1,sh);
    return sharpen;
}
public  Mat removeBlackout(Mat pic){
    try{
    Mat image = pic;

        double[][] sharpen_kernel = {{-1,-1,-1}, {-1,9.8,-1}, {-1,-1,-1}};
        Mat sh = new Mat(3,3, CvType.CV_16SC1);
        for(int i=0; i<3; i++){
            for(int j=0; j<3; j++){
                sh.put(i,j, sharpen_kernel[i][j]);
            }
        }
        Mat sharpen = new Mat();
        Imgproc.filter2D(image,sharpen, -1,sh);
        Imgproc.threshold(sharpen,sharpen,0,150,Imgproc.THRESH_TOZERO);
    return sharpen;
    }
    catch (Exception ex){
        return null;
    }

}

public  BufferedImage Mat2BufferedImage(Mat mat)
{
    try
    {
        return (BufferedImage) HighGui.toBufferedImage(mat);
    }
    catch (Exception ex){
        return null;
    }

}

public  Mat BufferedImage2Mat(BufferedImage im)  {
    try {
        byte[] pixels = ((DataBufferByte) im.getRaster().getDataBuffer())
                .getData();
        Mat image = new Mat(im.getHeight(), im.getWidth(), CvType.CV_8UC3);

        image.put(0, 0, pixels);
        return image;
    }
    catch (Exception ex){
        return null;
    }

}

}`

[huangh12]

@alexander-toschev I am curious whether your preprocessing steps improves the accuracy of QR code recognition or not.

[lessthanoptimal]

Did you run it through the same benchmark that was used to measure QR code performance on BoofCV's website? Trying to figure out where 30% came from.

Convolutions with custom kernels should be easy. FactoryConvolve is a good starting point. ThresholdImageOps for thresholding the image. Although it looks like your code might be convolving then thresholding the image. To do that in BoofCV you would need to do that in two steps.

Do you think it would be easier to try to get this working on a stand alone project initially?

[alexander-toschev]

@alexander-toschev I am curious whether your preprocessing steps improves the accuracy of QR code recognition or not.

Hi, yes, it gives me appox 30% more codes. The problem is that you need grab 1 image, and recognize it 3 times after each filter applied, than combine results. What i send is a part of my application. I will prepare this standalone later.

[alexander-toschev]

Did you run it through the same benchmark that was used to measure QR code performance on BoofCV's website? Trying to figure out where 30% came from.

Convolutions with custom kernels should be easy. FactoryConvolve is a good starting point. ThresholdImageOps for thresholding the image. Although it looks like your code might be convolving then thresholding the image. To do that in BoofCV you would need to do that in two steps.

Do you think it would be easier to try to get this working on a stand alone project initially?

Yes, i will do it.

[Herodin]

@alexander-toschev I thought about a similar approach but didn't have the time to sort it out, yet. Do you have any news about a standalone project?

[Herodin]

@lessthanoptimal

A lot of people want something that just works even if it's slow.

Exactly. I'm working on a batch QR reader that will extract QR data from hundreds of receipts. I don't mind if it takes 54 s or 67 s to process a folder with 100+ images as long as I don't have to typewrite the receipts manually (which would take hours if not days).