bug(barcode-scanning): multiple results while scanning same barcodes

[renekroll]

Plugin(s)

• [X] Barcode Scanning
• [ ] Face Detection
• [ ] Face Mesh Detection
• [ ] Selfie Segmentation
• [ ] Translation

Did you test the latest version?

• [X] I use the latest version

Platform(s)

• [X] Android
• [X] iOS
• [ ] Web

Current behavior

We migrated from the deprecated package to this package and run into to issues while scanning multiple codes.

When we scan multiple barcodes, the scanner recognizes several different results for the same barcodes.

https://github.com/capawesome-team/capacitor-mlkit/assets/38501755/4048446e-d30d-4fdd-811f-e10685c7f95b

Expected behavior

Deterministic recognition of scanned barcodes.

Reproduction

https://github.com

Steps to reproduce

Use the Scanner with multiple Codes or see our attached video.

Other information

Given Barcodes:

4058172100963 4058172309519 4058172310232

Detected Barcodes:

4058172100963 4058172309519 4058172310232 9061177310232 9061177309519 9061177100963 3051877309519 4058172310102 3051877310232

Capacitor doctor

Latest Dependencies:

@capacitor/cli: 5.5.1 @capacitor/core: 5.5.1 @capacitor/android: 5.5.1 @capacitor/ios: 5.5.1

Installed Dependencies:

@capacitor/cli: 5.5.1 @capacitor/core: 5.5.1 @capacitor/android: 5.5.1 @capacitor/ios: 5.5.1

Before submitting

• [X] I understand that incomplete issues (e.g. without reproduction) are closed.

[github-actions[bot]]

Please provide a Minimal, Reproducible Example using this template in a public GitHub repository so we can debug the issue.

[robingenz]

To debug this, I need to be able to reproduce it. I need photos with which I can reliably reproduce the problem in the demo app. However, I assume that the problem lies with the ML Kit SDK, as the plugin does not have much impact on performance and the final evaluation.

[Partfinger]

We are have the same problem. I have some experience with computer vision, and use an idea from it to solve this problem. I have next barcodes to scan: Default multiply scanning return next result: We also have mistakes. In order to solve it, i use vote system. With listener BarcodeScanner.addListener('barcodeScanned', ...) i cathcing all, that library can scan, and put scan result to array of objects {barcode: string, votes: number}. If some barcode adding to array first time, it have been voted once. Every scan of same barcode is incrementing voted. Then, we need votesThreshold variable, to filter barcodes. Mistakes will have no more 1-2 votes, correct barcodes - as many as possible, it depend of scanning time: You can see, mistakes have not enough votes to overcome threshold. You need find your threshold empirically. Low - give you a lot of mistakes, but fast. Big - correct result, but take a long time. In our case, threshold is 3-5 votes.

[stevedrew]

We also switched from the depreciated package and now getting lots of complaints about barcode 39 providing incorrect results. I can easily reproduce. Typically, when fails, the barcode image is not perfect, but good enough for any other scanner except this one.

Any update or suggestions?

Thanks.

[corpro]

Hi stevedrew,

Did you switch back to the deprecated package using AVFoundation and zxing? Any issues with Capacitor 5 or 6?

Thanks

[TobiTobsn007]

I currently only process codes if the same code has been scanned 4 times in a row. The scanner is fast enough for this quality check. It's a workaround. But maybe other scanners are doing this internally.

[corpro]

Hi Robin,

Do you think there is a fix in the near future if the problem lies within the ML Kit SDK?

Thanks!

[stevedrew]

@corpro we are in progress of reverting back to depreciated package as we also now have reports of people not being able to scan at all with some Samsung G20/21 devices and was working perfectly we previous depreciated package. Will report back later.

[robingenz]

@stevedrew Are you using the scan(...) or startScan(...) method?

[corpro]

@robingenz I have tested both methods in your demo app.

[corpro]

@robingenz these two screen shots are using the startscan() method scanning the same barcode 10 times (1st image is the correct barcode value) and the 2nd image showing a different value and corner points.

[stevedrew]

@corpro - We reverted back to previous depreciated package and all working well so far on both iOS and Android, our customers are happy again.

When we have more time, may try the "re-scan multiple times" workaround, but we also had the problem with some samsung devices not working at all with this new package.

[robingenz]

@Partfinger I tried to reproduce the issue with your examples but unfortunately without success. The captured barcode values are always correct.

As already mentioned in this comment, this plugin is not responsible for evaluating the barcodes but only passes on the results of the ML Kit Barcode Scanning SDK. This is therefore not a bug in the plugin. For this reason, I am closing this issue.

However, I understand the problem and would be willing to implement some kind of vote system. Therefore I have created the following PR: #129. Would someone who already had this problem be willing to test this PR to see if this feature would help?

npm i @capacitor-mlkit/barcode-scanning@5.3.0-dev.a2d5458.1708097442

The difference to an implementation at WebView level is that the startScan(...) method does not currently know the coordinates of a valid area and therefore includes all captured barcodes in the vote system.

[Partfinger]

@robingenz I should have described our problem in more detail so that you would understand the context. In general, we used some old scanner, but it was not supported after 2019, and after one of the capacitor updates, it stopped working on iPhones altogether. I installed the scanner from capacitor-community, it worked fine, but the following problems appeared. We complained that there are times when it is necessary to scan some barcodes, they may be in a pile, and after scanning it is not clear which barcode was scanned. I understood that such libraries are more focused on scanning a single barcode in the frame, but in our case this is more the exception than the rule. Then, we were given the task of scanning imei codes or serial numbers on product boxes. And the horror began, because the boxes, as a rule, have several barcodes in one place, and the workers had to cover up other barcodes with their fingers so that the desired one was scanned, and not the first one that the algorithms in the library liked. Then I found your library and it had the advantage of returning corner points. That way, I know where the barcode is on the screen and can filter out anything outside the scanning area. But this did not eliminate the problem that 80% of barcodes scanned via BarcodeScanner.addListener('barcodeScanned', ...) were usually not in the zone. I saw that, among other things, in the discussion of this repository, the question of somehow limiting the scanning area to ignore everything outside of it was raised, but it has not yet been implemented. But this problem was solved for the sake of it, while another one came out - imei codes, factory codes, serial numbers and similar barcodes - as a rule, long and complex, so problems like the scanner saw part of the barcode and interpreted it as the entire barcode when I scanned one and the same complex barcode - received different results, etc. That's when I came to the voting system, because it turned out that the probability of the same false activation of the scanner goes to zero, and the false results are usually different, so it makes sense to introduce the voting logic that I described in the post above. By the way, we complained that the scanner cannot scan white barcodes on a black background. I don't know, maybe you already know about it, but there is such a problem.

[robingenz]

@Partfinger Thanks for sharing!

I saw that, among other things, in the discussion of this repository, the question of somehow limiting the scanning area to ignore everything outside of it was raised, but it has not yet been implemented.

I just created a new issue to address this: #130

so problems like the scanner saw part of the barcode and interpreted it as the entire barcode when I scanned one and the same complex barcode - received different results

Okay, this is a very special case. Of course, the plugin can't do much here. The scan area should ideally not reach the edges of the screen, so that you can check whether the barcode is completely in the valid area.

By the way, we complained that the scanner cannot scan white barcodes on a black background. I don't know, maybe you already know about it, but there is such a problem.

I didn't know that. But the plugin won't be able to change anything here. Please report this in the ML Kit repository.