💭 Figuring out dimensions in `CodeScanner`

[rveltonCL]

Question

I am using the latest version of RN, expo and visions. But I am running into an issue trying to detect a barcode in a specific area of the screen.

My phones resolution: 926 x 428 Frame resolution: 1920 x 1080

My first thought was to simply take the ratio of the window and frame width and height, but it is not lining up 1:1 for what I am trying to do. Here is a sample of code: (please don't judge, just debugging a ton, lol)

const onCodeScanned = useCallback(async (codes: Code[], frame: CodeScannerFrame) => {
    // We should only have one QR code. If we have any value
    // other than one, exit out of this function.
    if (codes.length !== 1) {
        return;
    }
    // Setting the frame values depending how the framework
    // is rendering orientation. This should be able to
    // calculate it either way.
    const frameLongSide = Math.max(frame.width, frame.height);
    const frameShortSide = Math.min(frame.width, frame.height);

    // Grabbing the min and max values for the x and y
    // coordinates for the code corner values.
    const xValues = codes[0].corners?.map(p => p.x) || [];
    const yValues = codes[0].corners?.map(p => p.y) || [];
    const minX = Math.min(...xValues);
    const maxX = Math.max(...xValues);
    const minY = Math.min(...yValues);
    const maxY = Math.max(...yValues);

    // Setting the base window height and width values.
    const windowHeight = win.height;
    const windowWidth = win.width;

    // Setting the height and width ratios. Since we know the
    // app has to be in portrait mode, we know which values
    // to divide to get to the ratios.
    const heightRatio = windowHeight / frameLongSide;
    const widthRatio = windowWidth / frameShortSide;

    const scanAreaDifference = win.width * 0.1;
    const scanAreaXMin = windowWidth - (offsetWidth + scanAreaWidth);
    const scanAreaXMax = scanAreaXMin + scanAreaDifference;
    const scanAreaYMin = (offsetHeight + scanAreaHeight) - scanAreaDifference;
    const scanAreaYMax = (offsetHeight + scanAreaHeight);

    const calculatedXMin = (minY * widthRatio);
    const calculatedXMax = (maxY * widthRatio);
    const calculatedYMin = (minX * heightRatio);
    const calculatedYMax = (maxX * heightRatio);

    console.log(isCapturing);
    if (calculatedXMin > scanAreaXMin &&
        calculatedXMax < scanAreaXMax &&
        calculatedYMin > scanAreaYMin &&
        calculatedYMax < scanAreaYMax &&
        !isCapturing) {
        console.log('WE HAVE LIFT OFF!');

        setIsCapturing(true);
        setIsPhotoActive(true);
        console.log(isPhotoActive);
        // await takePhoto();
    }

    console.log(Dimensions.get('window').width, Dimensions.get('window').height);
    console.log('codes', JSON.stringify(codes));
    console.log('frame', JSON.stringify(frame));
    console.log(minX, maxX, minY, maxY);
    console.log('Scan Area Difference:', scanAreaDifference);
    console.log('Ratios:', heightRatio, widthRatio);
    console.log('Calculated X Min:', calculatedXMin);
    console.log('Calculated X Max:', calculatedXMax);
    console.log('Calculated Y Min:', calculatedYMin);
    console.log('Calculated Y Max:', calculatedYMax);
    console.log('Scan Area X Min:', scanAreaXMin);
    console.log('Scan Area X Max:', scanAreaXMax);
    console.log('Scan Area Y Min:', scanAreaYMin);
    console.log('Scan Area Y Max:', scanAreaYMax);
}, [
    offsetWidth,
    offsetHeight,
    scanAreaWidth,
    scanAreaHeight
]);

And here is the output in the console log:

LOG 428 926 LOG codes [{"type":"qr","frame":{"x":1350.5323791503906,"height":75.73731601238251,"width":75.75244903564453,"y":199.54500496387482},"corners":[{"x":1351.094835691224,"y":275.2823113601515},{"x":1426.2848074981866,"y":273.77842710326416},{"x":1424.5970270713685,"y":199.5450096337763},{"x":1350.532352809027,"y":201.02617018822104}],"value":"239551 1"}] LOG frame {"width":1920,"height":1080} LOG 1350.532352809027 1426.2848074981866 199.5450096337763 275.2823113601515 LOG Scan Area Difference: 42.800000000000004 LOG Ratios: 0.4822916666666667 0.3962962962962963 LOG Calculated X Min: 79.07894826227431 LOG Calculated X Max: 109.09336042791189 LOG Calculated Y Min: 651.3504993235204 LOG Calculated Y Max: 687.8852769496463 LOG Scan Area X Min: 30.33335304260254 LOG Scan Area X Max: 73.13335304260255 LOG Scan Area Y Min: 677.5332977294922 LOG Scan Area Y Max: 720.3332977294922

The "calculated" values are from the barcode frame, and I just can not get them to line up with the scan area from one of the view elements. I feel like this is something simple that I am just missing, but can't seem to figure it out. Any help would be greatly appreciated. Thanks.

What I tried

No response

VisionCamera Version

3.8.2

Additional information

• [X] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar questions in the issues page as well as in the discussions page and found none.

[mrousavy]

Wow this looks complicated lol - hm are you sure the Code object doesn't have anything related in there?

[brunoobatista]

I am facing the same difficulty; I want to draw the location of the QR Code on the screen as it appears, but there is this discrepancy with the size data of the frame and the screen. Here is an example of Code:

{"corners": [{"x": 467, "y": 314}, {"x": 751, "y": 311}, {"x": 749, "y": 609}, {"x": 458, "y": 601}], "frame": {"height": 298, "width": 293, "x": 458, "y": 311}, "type": "qr", "value": "some_url"}

Here is my screen settings mobile Android:

width: 360
height: 740
scale: 3
fontScale: 1

I've already tried scaling division, attempted to come up with anything to make the Code data resemble the values of the device. However, without success. I can even draw the rectangle on the screen using scaling to try to match the frame dimensions with the screen dimensions. However, the rectangle always ends up out of position.

[mrousavy]

Did you take the sensor's natural orientation into account? device.sensorOrientation

Again, Orientation isn't implemented yet as it is tracked here: https://github.com/mrousavy/react-native-vision-camera/issues/1891

Once that is implemented, everything that should be in portrait will be in portrait and there's helpers to convert coordinates.

[brunoobatista]

Ah, I apologize... I understood a bit better how it works and saw that it is something complex to resolve. I will try to implement this QR Code detection using the frameProcessor.

[RongDuJiKsp]

I also encountered this puzzling problem. I found that the data given by Codescanner throughout the afternoon and one night was very different from the thinking of conventional people. In the data given by CodeScanner, x is the distance from the top of the scanning box, and y is the distance from the right side of the scanning box This design is very puzzling. In the thinking of a general front -end programmer, X should be a distance from the left, and Y should be the distance from the top Similarly, I found that the length and width in the Result obtained by Oncodescanner seemed to be inconsistent with the length and width of the front -end programmer. When the style is long, the left and right are long, and the upper and lower are wide, which seems to cause misunderstandings I hope to present this in the documentation I used the code below, it works well

                <View className={"w-full h-full absolute left-0 top-0"}>
                    {toFocusingCodeScannerResult?.codes.map((code, index) => {
                        if (code.frame === undefined) return <></>
                        return <Text key={"scanner-focus-no" + index} style={{
                            left: `${100 - (code.frame.y + code.frame.width/2) / toFocusingCodeScannerResult?.frame.height * 100}%`,
                            top: `${(code.frame.x + code.frame.height / 2) / toFocusingCodeScannerResult?.frame.width * 100}%`,
                        }} className={"text-green-600 relative"}>[[+]]</Text>;
                    })}
                </View>

[RongDuJiKsp]

https://github.com/mrousavy/react-native-vision-camera/assets/110959752/1826d7ce-9076-432f-8353-5e89c0d9b1f8

@rveltonCL @brunoobatista Try this method? It works well

[metrix-hu]

@brunoobatista I have a patch file for sensor orientation in CodeScannerPipeline for Android. If you are still interested I can post it here later today. Apart from that issue its really straightforward with the current API how to solve the scaling. I have a working example with an RN Skia view on top of the camera.

[mrousavy]

Nice work @RongDuJiKsp 👍

[brunoobatista]

freecompress-Screenrecording_20240204_122414.mp4 @rveltonCL @brunoobatista Try this method? It works well

@RongDuJiKsp I'll need to digest your explanation; it left me a bit confused, haha. But I'll try to apply your advice to my situation and see how it goes. Thanks for your attention.

@brunoobatista I have a patch file for sensor orientation in CodeScannerPipeline for Android. If you are still interested I can post it here later today. Apart from that issue its really straightforward with the current API how to solve the scaling. I have a working example with an RN Skia view on top of the camera.

@metrix-hu I would like to see your example, it would be of great help to me.

[metrix-hu]

@brunoobatista Here is my patch file: react-native-vision-camera+3.8.2.patch

[metrix-hu]

@brunoobatista Also here is my complete component code:

import React, {
    forwardRef,
    useCallback,
    useEffect,
    useImperativeHandle,
    useMemo,
    useRef,
    useState,
} from 'react';
import {StyleSheet, Text, View} from 'react-native';
import {
    Camera,
    Code,
    CodeScannerFrame,
    useCameraDevice,
    useCodeScanner,
} from 'react-native-vision-camera';
import {useSharedValue, useWorklet, Worklets} from 'react-native-worklets-core';
import {
    createPicture,
    PaintStyle,
    Skia,
    SkiaPictureView,
    SkiaView,
    TileMode,
} from '@shopify/react-native-skia';

import {useAppStyles, useSkiaPaint} from '../utils/Hooks';

export type CodeChecker = (code: string) => boolean;

export function useCodeChecker(cb: CodeChecker, deps: React.DependencyList) {
    return useCallback(cb, [cb, ...deps]);
}

export type CodeProcessor = (code: string) => void;

export function useCodeProcessor(cb: CodeProcessor, deps: React.DependencyList) {
    return useCallback(cb, [cb, ...deps]);
}

export interface ScannedCode extends Code {
    success?: boolean;
    [key: string]: any;
}

export interface ScannerCameraProps {
    codeChecker?: CodeChecker;
    codeProcessor?: CodeProcessor;
    children?: React.ReactNode;
}

export interface ScannerCamera {
    stop(): void;
    resume(): void;
}

const cyan = Skia.Color(0xff00ffff);
const red = Skia.Color(0xffff0000);
const green = Skia.Color(0xff00ff00);

export const ScannerCamera = forwardRef<ScannerCamera, ScannerCameraProps>((props, ref) => {
    const camera = useRef<Camera>(null);
    const device = useCameraDevice('back');
    const styles = useAppStyles();

    const paint = useSkiaPaint();

    const [isActive, setActive] = useState(false);
    const [codes, setCodes] = useState<ScannedCode[]>([]);
    const scannerFrame = useSharedValue<CodeScannerFrame>(null);
    const successCode = useSharedValue<Code>(null);
    const setCodesJs = Worklets.createRunInJsFn(setCodes);

    const codeScanner = useCodeScanner({
        codeTypes: ['qr'],
        onCodeScanned: (scanned, frame) => {
            if (!isActive) {
                return;
            }
            setCodesJs(
                scanned.map(code => {
                    const success = props.codeChecker ? props.codeChecker(code.value) : null;
                    if (!successCode.value && success) {
                        successCode.value = code;
                    }
                    return {
                        success,
                        ...code,
                    };
                }),
            );
            scannerFrame.value = frame;
        },
    });
    const [rect, setRect] = useState(Skia.XYWHRect(0, 0, 100, 100));

    const picture = useMemo(() => {
        return createPicture(rect, canvas => {
            if (successCode.value) {
                const value = successCode.value.value;
                // Immediately stop the camera if we have a success code
                setActive(false);
                successCode.value = null;
                if (props.codeProcessor) {
                    // Call codeProcessor after a little timeout
                    setTimeout(() => {
                        props.codeProcessor(value);
                    }, 500);
                }
                return;
            }

            const frame = scannerFrame.value;

            if (!frame) {
                return;
            }

            const crx = rect.width / frame.width;
            const cry = rect.height / frame.height;
            const scale = Math.max(crx, cry);
            const tx = (rect.width - frame.width * scale) / 2;
            const ty = (rect.height - frame.height * scale) / 2;

            paint.setStyle(PaintStyle.Stroke);
            codes.map(code => {
                let color = cyan;

                if (typeof code.success === 'boolean') {
                    if (code.success) {
                        color = green;
                    } else {
                        color = red;
                    }
                }

                const corners = Object.values(code.corners);

                corners.forEach((corner: any, ix: any) => {
                    // Calculate the points of the line
                    const next = corners[(ix + 1) % 4];
                    const cx = corner.x * scale + tx;
                    const cy = corner.y * scale + ty;
                    const nx = next.x * scale + tx;
                    const ny = next.y * scale + ty;

                    paint.setColor(color);
                    paint.setStrokeWidth(2);
                    canvas.drawLine(cx, cy, nx, ny, paint);
                });
            });
        });
    }, [codes, paint, props, rect, scannerFrame.value, successCode]);

    useEffect(() => {
        Camera.requestCameraPermission().then(permission => {
            const res = permission as string;
            setActive(res === 'granted' || res === 'authorized');
        });
        return () => {
            setActive(false);
        };
    }, []);

    useImperativeHandle(ref, () => ({
        resume: () => {
            setActive(true);
        },
        stop: () => {
            setActive(false);
        },
    }));

    if (device == null) {
        return (
            <View style={styles.roundContainer}>
                <Text style={styles.screenTitle}>No camera available</Text>
            </View>
        );
    }

    return (
        <View style={styles.roundContainer}>
            <Camera
                ref={camera}
                style={StyleSheet.absoluteFill}
                device={device}
                isActive={isActive}
                enableZoomGesture={true}
                photo={true}
                codeScanner={codeScanner}
            />
            <SkiaPictureView
                onLayout={ev => {
                    const layout = ev.nativeEvent.layout;
                    setRect(Skia.XYWHRect(0, 0, layout.width, layout.height));
                }}
                style={StyleSheet.absoluteFill}
                pointerEvents='none'
                mode='continuous'
                picture={picture}
            />
            {props.children}
        </View>
    );
});

[oscar-b]

Is this really needed/expected? I thought the onCodeScanned callback was already in JS land?

const setCodesJs = Worklets.createRunInJsFn(setCodes);

[mrousavy]

Yes, @oscar-b you are right - CodeScanner is already in JS, there are no Worklets involved in the built-in CodeScanner.

[metrix-hu]

@oscar-b Nice catch, I will improve that too.

[johnernest02-automanager]

Hi @metrix-hu I'd like to ask what is the utils/Hooks? I am trying out your solution as it seems like it is the solution to my problem as well. Can you share them with me please?

[pke]

Nice work @RongDuJiKsp 👍

shouldn't the lib provide this normalised already for the user? The RN coordinate system is x,y being top/left and the Android Frame coordinates should match this and not put the burden on every single API user to convert the values (no docs provided!)

[mrousavy]

shouldn't the lib provide this normalised already for the user?

Yea agreed, I guess it should. If anyone wants to contribute and send a PR, I'd appreciate it. Otherwise I will add this when I have some free time (never)

[mrousavy]

Otherwise I will add this when I have some free time (never)

(just kidding, but it will take maybe in a month or so because of V4 and the 3D library rn)

[pke]

Wanted to report the values for iOS are also off. The frame coordinates can not be used in the RN world to place content on the screen with absolute coordinates.

[shacharcohen1997]

freecompress-Screenrecording_20240204_122414.mp4 @rveltonCL @brunoobatista Try this method? It works well

can you share the code of the scanning ?

[shacharcohen1997]

@brunoobatista Here is my patch file: react-native-vision-camera+3.8.2.patch

can you share the ../utils/Hooks please?

[metrix-hu]

@shacharcohen1997 Here is how the relevant hooks looks like:

export function useAppStyles(): AppStyles {
    return useAppThemeProp(theme => theme.styles);
}
export function useSkiaPaint(blendMode: BlendMode = BlendMode.SrcOver): SkPaint {
    return useMemo(() => {
        const paint = Skia.Paint();
        paint.setBlendMode(blendMode);
        return paint;
    }, [blendMode]);
}

The first one just returns a simple styles object is not rocket science. I wont share more. The second uses imports from "@shopify/react-native-skia"

[JshGrn]

I am trying to restrict scanning of QR codes to within a box I have placed on the screen, I have the x and y positions of the box along with the width and the height. I cannot make sense of the corners given by the code, the docs are most unhelpful here.

Am I correct in thinking that the corners are not the actual corners and are the distance from the left and right, almost like the middle of each? What order are the corners?

I simply want to write some code which says is the QR code within this box, if yes, scan it... Would anyone be able to guide me in a direction to look at? I have opened a thread on Discord support here (the discord for this package) and have had zero feedback or input. I believe @metrix-hu has this sussed out and perhaps a couple of others however nobody else has been as open with their solution.

[spsaucier]

I had this working a few months ago, but it stopped working in a release since then (center box shrunk & moved for some phones), so I've had it disabled. Not sure there is a solution that works on all phones right now.

[JshGrn]

I had this working a few months ago, but it stopped working in a release since then (center box shrunk & moved for some phones), so I've had it disabled. Not sure there is a solution that works on all phones right now.

I actually only need it to work on one specific device model, I will take a look at your code to see how you are achieving it, my issue is some of the corners have a negative position which given they are relative to the camera preview.... I am confused about. It would be really good to understand more about these corners in the docs and why they may be negative values.

Thx for the link 👍

[mrousavy]

VisionCamera exposes device.sensorOrientation, maybe you need to rotate by that.

Either way, I don't have the free time to fix this right now, maybe I'll look into this in the future, we'll see.

[JshGrn]

I actually realised this was fixed in a PR you merged which better tracked the location of the QR code, from this I was able to correctly track the QR code and its bounds on the screen. So I have been able to do what I needed to now after that update :).

Thank you.

[mrousavy]

Awesome! 👍