🐛 Camera preview is not showing the full content

[xulihang]

What's happening?

I am building a document scanning app. It runs the frame processor to detect documents and take a photo after the detection is steady.

Check out the following video and we can see that although I've set the aspect ratio of both the video and the photo to 16/9, the camera preview is not showing the full content as in the image taken.

https://github.com/user-attachments/assets/932d1a31-58cd-452c-9734-bc9b2f920fa5

Reproduceable Code

const device = useCameraDevice("back");
const cameraFormat = useCameraFormat(device, [
  { videoResolution: { width: 1280, height: 720 } },
  { photoAspectRatio: 16/9 },
  { videoAspectRatio: 16/9 },
  { fps: 60 }
])
<Camera
  style={StyleSheet.absoluteFill}
  ref={camera}
  isActive={isActive}
  device={device}
  photo={true}
  format={cameraFormat}
  frameProcessor={frameProcessor}
  resizeMode='contain'
/>

Relevant log output

LOG  detect frame
 LOG  1280 x 720 yuv Frame
 LOG  {"height": 4096, "isMirrored": false, "isRawPhoto": false, "orientation": "portrait", "path": "/data/user/0/com.documentscanner/cache/mrousavy5124308965332285767.jpg", "width": 2304}
 LOG  {"height": 4096, "isMirrored": false, "isRawPhoto": false, "orientation": "portrait", "path": "/data/user/0/com.documentscanner/cache/mrousavy5124308965332285767.jpg", "width": 2304}

Camera Device

{
  "formats": [],
  "hardwareLevel": "full",
  "hasFlash": true,
  "hasTorch": true,
  "id": "0",
  "isMultiCam": false,
  "maxExposure": 18,
  "maxZoom": 10,
  "minExposure": -18,
  "minFocusDistance": 5,
  "minZoom": 1,
  "name": "0 (BACK) androidx.camera.camera2",
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "position": "back",
  "sensorOrientation": "landscape-left",
  "supportsFocus": true,
  "supportsLowLightBoost": false,
  "supportsRawCapture": false
}

Device

OPPO A72 5G

VisionCamera Version

4.5.3

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

• [ ] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here! 🍻

Looks like you have a valid issue with ze camera preview not displaying correctly. Vielen Dank for providing ze details.

Since you mentioned you cannot reproduce ze issue in ze Example app, could you try a few things? First, make sure your test environment matches ze one where you see ze issue. Also, consider checking if there are any device-specific limitations with your OPPO A72 5G.

If ze issue persists, could you try gathering more detailed logs? For Android, you can use adb logcat to capture ze runtime logs when you run your app. This will help mrousavy diagnose ze problem better.

Let us know how it goes!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[xulihang]

Because of this problem, I cannot draw the overlay accurately as well.

[xulihang]

If we use Skia Frame Processor, the camera preview is correct.

However, since it has to convert YUV to RGB on Android, the performance is really bad.

https://github.com/user-attachments/assets/cfc2d0c2-a75a-4362-b3e9-ceff383bd8fe

[xulihang]

I updated skia to support the yuv format: https://github.com/Shopify/react-native-skia/pull/2357

But if I use a frame processor, the FPS is not good.

So I ended up using skia frame processor to draw the camera preview and using react-native-svg to draw the overlay.

https://github.com/user-attachments/assets/798dd22e-8161-46d6-9ad4-4081e5924768

[mrousavy]

This is because Preview and Frame Processor are two separate output streams. Preview is a PREVIEW stream, Frame Processor is a VIDEO stream. They are not guaranteed to match. Skia Frame Processors are guaranteed to match, as there is no PREVIEW stream - both preview and frame processor are running on the same VIDEO stream.

Sorry, but there is nothing I can do about this. PREVIEW and VIDEO are simply not guaranteed to match on Android. Skia FPs are the way to go.