bug: iOS Corner points detection issue

[Melynt3]

Plugin(s)

• [x] Barcode Scanning
• [ ] Face Detection
• [ ] Face Mesh Detection
• [ ] Selfie Segmentation
• [ ] Translation

Version

5.3.0-dev.966ebeb.1703101506

Platform(s)

• [ ] Android
• [X] iOS
• [ ] Web

Current behavior

I have problem trying to scan. The corner point problem occurs only on iOS device in my case.

Can't understand why iOS coords are different on every angle im attempting to scan. Is there a way to lock it somehow? I seems like i can get scan go through on lucky strike.

Android works fine for me.

my scan log:

{"rawValue":"ASDSADASDAS","bytes":[71,80,76,50,50,48,53,48,53,50,56,50,57],"cornerPoints":[[686,973],[681,1069],[165,1094],[174,1008]],"format":"CODE_128","valueType":"TEXT","displayValue":"ASDSADASDAS"}
⚡️  [log] - Detection Corner Points: [[57,519.7821655273438],[357,519.7821655273438],[357,644.7821655273438],[57,644.7821655273438]]
⚡️  [log] - Current Corner Points: [[686,973],[681,1069],[165,1094],[174,1008]]
⚡️  [log] - Conditions not met. Returning...

It makes problem because as u can see in logs scan never getting the value and enter never ending loop, when im commenting corner detection scanner works fine.

Would be greatfull for some tips/help

similar case: there is opened issue on this case. https://github.com/capawesome-team/capacitor-mlkit/issues/115

Expected behavior

I expect the success scan when catch barcode in my scan box.

Reproduction

none

Steps to reproduce

1. Add some logs to see detection coords and current cords
2. Run demo barcode scanner on iOS device
3. Try to scan from different angles

Other information

No response

Capacitor doctor

Latest Dependencies:

@capacitor/cli: 5.7.0 @capacitor/core: 5.7.0 @capacitor/android: 5.7.0 @capacitor/ios: 5.7.0

Installed Dependencies:

@capacitor/core: 5.3.0 @capacitor/ios: 5.0.5 @capacitor/cli: 5.0.5 @capacitor/android: 5.3.0

[success] iOS looking great! 👌

Before submitting

• [X] I have read and followed the bug report guidelines.
• [X] I have attached links to possibly related issues and discussions.
• [X] I understand that incomplete issues (e.g. without reproduction) are closed.

[robingenz]

Please create a screen recording and show the problem there.

[Melynt3]

the startScan code:

    private async startScan(): Promise<void> {
        document.querySelector('body')?.classList.add('barcode-scanning-active');

        const options: StartScanOptions = {
            formats: this.formats,
            lensFacing: this.lensFacing
        };

        const squareElementBoundingClientRect =
            this.squareElement?.nativeElement.getBoundingClientRect();
        const scaledRect = squareElementBoundingClientRect
            ? {
                left: squareElementBoundingClientRect.left * window.devicePixelRatio,
                right: squareElementBoundingClientRect.right * window.devicePixelRatio,
                top: squareElementBoundingClientRect.top * window.devicePixelRatio,
                bottom: squareElementBoundingClientRect.bottom * window.devicePixelRatio,
                width: squareElementBoundingClientRect.width * window.devicePixelRatio,
                height: squareElementBoundingClientRect.height * window.devicePixelRatio
            }
            : undefined;

        console.log('Scaled Rect:', scaledRect);

        const detectionCornerPoints = scaledRect
            ? [
                [scaledRect.left, scaledRect.top],
                [scaledRect.left + scaledRect.width, scaledRect.top],
                [
                    scaledRect.left + scaledRect.width,
                    scaledRect.top + scaledRect.height
                ],
                [scaledRect.left, scaledRect.top + scaledRect.height]
            ]
            : undefined;

        const listener = await BarcodeScanner.addListener(
            'barcodeScanned',
            async (event) => {
                this.ngZone.run(() => {
                    const cornerPoints = event.barcode.cornerPoints;
                    console.log('Scanned Corner Points:', cornerPoints);
                    console.log('Detection Corner Points:', detectionCornerPoints);

                    if (detectionCornerPoints && cornerPoints) {
                        if (
                            detectionCornerPoints[0][0] > cornerPoints[0][0] ||
                            detectionCornerPoints[0][1] > cornerPoints[0][1] ||
                            detectionCornerPoints[1][0] < cornerPoints[1][0] ||
                            detectionCornerPoints[1][1] > cornerPoints[1][1] ||
                            detectionCornerPoints[2][0] < cornerPoints[2][0] ||
                            detectionCornerPoints[2][1] < cornerPoints[2][1] ||
                            detectionCornerPoints[3][0] > cornerPoints[3][0] ||
                            detectionCornerPoints[3][1] < cornerPoints[3][1]
                        ) {
                            console.log('Condition not met, returning.');
                            return;
                        }
                    }
                    listener.remove();
                    this.closeModal(event.barcode);
                });
            }
        );

        console.log('Starting Barcode Scan...');
        await BarcodeScanner.startScan(options);
    }

Will that be enough? In the last seconds of video im getting success, because my phone was in vertical position, only in this case it goes through. https://github.com/capawesome-team/capacitor-mlkit/assets/45972864/83f9be6b-6fb3-4a7a-aaff-ede28c793637

It clearly see the barcode instantly but can get this scan as well from the scan box, my guess i scalling

[robingenz]

capawesome-team/capacitor-mlkit/assets/45972864/83f9be6b-6fb3-4a7a-aaff-ede28c793637

I meant a screen recording of the device.

[Melynt3]

the screen is just scanning modal pointed on my barcode :D I just changed angle of camera.

There's no info on the screen or something

[Melynt3]

Do you need any screen record? I mean there's nothing to see there, just view of phone trying to scan barcode from different angle

[robingenz]

Yes, please upload a screen recording of the device.

[Melynt3]

This is how it works right now, did not noticed that video bricked. But as u can see the scan goes thorugh after changing angle

https://github.com/capawesome-team/capacitor-mlkit/assets/45972864/514a1b3e-fb2c-4d55-8818-2a51fc1b041d

[robingenz]

Thank you for the video! Can you share an image file with the barcode so I can test it myself?

Have you tried holding the camera further away, regardless of the angle?

[Melynt3]

Thank you for the video! Can you share an image file with the barcode so I can test it myself?

Sure heres the img:

Have you tried holding the camera further away, regardless of the angle?

Ye i tired it from bigger distance and many other ways. The first vid that i send shows the diff in coords

---

I was thinking of flat workaround, in theory we could set the Y axis to the middle of our modal and the gave him eg. 200 height from there (100 top, 100 bottom) with 100% width. Which would make a box in the center of page with 200x100%

I think it would be nice to be able to set beacuse some devices may have problems with window scaling, it could prevent form scanning with whole modal when few barcodes would be in area, what do u think about it?

[robingenz]

I was thinking of flat workaround, in theory we could set the Y axis to the middle of our modal and the gave him eg. 200 height from there (100 top, 100 bottom) with 100% width. Which would make a box in the center of page with 200x100%

I think it would be nice to be able to set beacuse some devices may have problems with window scaling, it could prevent form scanning with whole modal when few barcodes would be in area, what do u think about it?

I'm not quite sure what you mean. You can already limit the area via JS. Do you want this to happen on the native level or how?

[Melynt3]

I was thinking of flat workaround, in theory we could set the Y axis to the middle of our modal and the gave him eg. 200 height from there (100 top, 100 bottom) with 100% width. Which would make a box in the center of page with 200x100% I think it would be nice to be able to set beacuse some devices may have problems with window scaling, it could prevent form scanning with whole modal when few barcodes would be in area, what do u think about it?

I'm not quite sure what you mean. You can already limit the area via JS. Do you want this to happen on the native level or how?

I wrote it a bit wrong, I meant to mark the scanning field in the center, without using corner points. flat area in js. And I was wondering if it had any right to exist

[robingenz]

There are users who want this area above or below. Not everyone wants it in the middle.

[Melynt3]

There are users who want this area above or below. Not everyone wants it in the middle.

Ye, and i respect that, but was asking certainly about my case, it could skip the problem with part where code does not meet conditions, which bricks my app now somehow, because it sees the barcode immediately but detectionCornerPoints && cornerPoints not match.

 if (detectionCornerPoints && cornerPoints) {
                        if (
                            detectionCornerPoints[0][0] > cornerPoints[0][0] ||
                            detectionCornerPoints[0][1] > cornerPoints[0][1] ||
                            detectionCornerPoints[1][0] < cornerPoints[1][0] ||
                            detectionCornerPoints[1][1] > cornerPoints[1][1] ||
                            detectionCornerPoints[2][0] < cornerPoints[2][0] ||
                            detectionCornerPoints[2][1] < cornerPoints[2][1] ||
                            detectionCornerPoints[3][0] > cornerPoints[3][0] ||
                            detectionCornerPoints[3][1] < cornerPoints[3][1]
                        ) {
                            console.log('Condition not met, returning.');  <------- this part
                            return;
                        }
                    }
                    listener.remove(); 
                    this.closeModal(event.barcode);

[robingenz]

I just fixed some corner points calculation issues in #127. Please give this dev version a try and let me know if it resolves your issues:

npm i @capacitor-mlkit/barcode-scanning@5.3.0-dev.b1cf5ce.1708071895

[Melynt3]

ill check it right away, thanks.

[Melynt3]

Sorry for late response. I have installed this version but it still can go through while scanning eg. from the table it catches like on the video i have send u above

{"barcode":{"bytes":[71,80,72,50,51,48,52,49,49,49,50,54],"rawValue":"GPH230411126","format":"CODE_128","displayValue":"GPH230411126","valueType":"TEXT","cornerPoints":[[257,886],[673,821],[684,918],[268,968]]}}
⚡️  [log] - Scanned Corner Points: [[257,886],[673,821],[684,918],[268,968]]
⚡️  [log] - Detection Corner Points: [[114,991.6085815429688],[714,991.6085815429688],[714,1241.608642578125],[114,1241.608642578125]]
⚡️  [log] - Condition not met, returning.

[robingenz]

In this case you need to debug it yourself. I cannot reproduce any issues with latest version. Feel free to create a PR with a fix.

[Melynt3]

seems like detection corner points are rotated 90 degrees, placed them from 3 position on chart, any clue with that?

[robingenz]

Are you sure you use the latest version?

[Melynt3]

Are you sure you use the latest version?

@capacitor-mlkit/barcode-scanning@5.3.0-dev.b1cf5ce.1708071895 this one

[robingenz]

Please give it a try with the latest version (5.4.0):

npm i @capacitor-mlkit/barcode-scanning@latest

[Melynt3]

Please give it a try with the latest version (5.4.0):

npm i @capacitor-mlkit/barcode-scanning@latest

Hm i tested it and it seems to work, iOS is still working poorly, u have to move it some much to catch it, which gives feeling like it slowed.

When I hover over the barcode, I see the logs returning with results not being in scanning box, but after a while in the same position suddenly the code is treated as being in the field and the scan passes. Which is a bit strange.

Whole process of scanning works bit slower overally on iOS and Android

[robingenz]

Thanks for testing. That's weird. It works without problems for me. I recently shared a screen recording on X: https://x.com/robin_genz/status/1758536783414071431?s=46&t=k9Xz6D9ZXLbWibNDwFk3nQ

As I said, feel free to debug the issue on your end. PRs are welcome!

[Melynt3]

ill test it more tomorrow, it may be affected by camera losing focus from the center, and the code may be unreadable? Or, the structure of html moves my scan box somehow, cus i added header and footer which made ion-content little smaller.

Is this topic in ur schedule #119

missclick with close as completed

[robingenz]

Is this topic in ur schedule #119

It's still on my todo list but I don't have any ETA. Next, I'm working on the Capacitor 6 migration.

[Melynt3]

Almost got it works fine, but one thing is bugging me. The scanning box catches 40-50px below the box and the opposite is at the top, It kind of went downhill a bit for those (bottom part of scan box catches well, and little below him, top part is somehow turned off :D).

Thats the last problem im facing, did u set centering mechanism, or added some values to this scan box part in 5.4?

My html is little different, do you see any issue in mine?

 <ion-header class="shader">
            <ion-toolbar class="ios">
                <ion-icon
                        (click)="closeModal()"
                        class="td-cursor-pointer td-position-relative td-right-0 td-z-index ion-padding ion-margin"
                        name="close"
                        size="large"
                        slot="start"
                        color="light">
                </ion-icon>
                <ion-icon
                        (click)="toggleTorch()"
                        class="td-cursor-pointer td-position-relative td-z-index ion-padding ion-margin"
                        [name]="torchActive ? 'flashlight' : 'flashlight-outline'"
                        size="large"
                        slot="end"
                        [color]="torchActive ? 'success' : 'light'">
                </ion-icon>
            </ion-toolbar>
        </ion-header>

        <ion-content>
            <div #scanBox class="scan-box">
                <p></p>
            </div>

            <ion-fab slot="fixed" vertical="bottom" horizontal="end">

                <ion-fab-button
                        class="in"
                        type="button"
                        size="large"
                        (click)="zoomIn()"
                        (touchstart)="startZoomIn()"
                        (touchend)="stopZoom()">
                    <ion-icon name="add"></ion-icon>
                </ion-fab-button>

                <ion-fab-button
                        class="out"
                        type="button"
                        size="large"
                        (click)="zoomOut()"
                        (touchstart)="startZoomOut()"
                        (touchend)="stopZoom()">
                    <ion-icon name="remove"></ion-icon>
                </ion-fab-button>
            </ion-fab>
            <div class="zoom-ratio-wrapper">
            <p class="ion-text-center app-mt-0">
                {{ zoomRatio >= 1 ? Math.round(zoomRatio * 100) + '%' : '100%' }}
            </p>
        </div>
        </ion-content>
    `,
    styles: [
        `
          ion-content {
            --background: transparent;
          }

          ion-fab-button {
            margin-bottom: 16px;
            width: 78px;
            height: 78px;
            --box-shadow: 0px 1px 2px 0px rgba(0, 0, 0, 0.3), 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
            --color: #fff;
            --background: var(--ion-color-dark) !important;
            opacity: 0.6;

          }

          ion-modal {
            --border-radius: 0;
          }

          .shader {
            background-color: rgba(0, 0, 0, 0.5) !important;
            box-shadow: none !important;
            z-index: 0;
          }

          .ios ion-toolbar {
            --background: transparent;
            --ion-color-base: transparent !important;
            --border-width: 0;
          }

          ion-button {
            margin-inline: unset
          }

          .box {
            box-shadow: 0 0 0 100vmax rgba(0, 0, 0, 0.6);
            position: absolute;
            width: 330px;
            height: 100px;
            left: 50%;
            top: 50%;
            transform: translate(-50%, -50%);
          }

          .scan-box:before, .scan-box:after, .scan-box > :first-child:before, .scan-box > :first-child:after {
            position: absolute;
            width: 22px;
            height: 22px;
            border-color: var(--ion-color-light);
            border-style: solid;
            content: '';
          }

          .scan-box:before {
            top: -8px;
            left: -8px;
            border-width: 3px 0 0 3px
          }

          .scan-box:after {
            z-index: -1;
            top: -8px;
            right: -8px;
            border-width: 3px 3px 0 0
          }

          .scan-box > :first-child:before {
            bottom: -8px;
            right: -8px;
            border-width: 0 3px 3px 0
          }

          .scan-box > :first-child:after {
            bottom: -8px;
            left: -8px;
            border-width: 0 0 3px 3px
          }

          .zoom-ratio-wrapper {
            position: absolute;
            left: 50%;
            bottom: 16px;
            transform: translateX(-50%);
            width: 50%;
          }

[robingenz]

No, your html seems to be okay.

[Melynt3]

ill record u tomorrow in the morning what it looks like in action, cant understand that. Maybe u might be able to localize the issue

[github-actions[bot]]

It looks like there hasn't been a reply in 30 days, so I'm closing this issue.