search

mrousavy / react-native-vision-camera

📸 A powerful, high-performance React Native Camera library.
https://react-native-vision-camera.com
MIT License
7.38k stars 1.08k forks source link

🐛 Frame processor + take photo causes crashes #2141

Closed bglgwyng closed 8 months ago

bglgwyng commented 11 months ago

What's happening?

Calling camera.takePhoto after switching the device causes crashes when the frame processor is performing reads on the buffer. This behavior was noticed within my application, which involves performing ML tasks on frames. If you run the null frame processor which does nothing within it, camera.takePhoto works well. Also, running the example frame processor plugin doesn't cause crashes.

  1. Run the example app with the following frame processor code
  2. Switch the device from back to front
  3. Call takePhoto

Reproduceable Code

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'

  // Commenting this out makes crashes not happen
  frame.toArrayBuffer()
}, [])

Relevant log output

2023-11-08 20:24:16.648 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  Camera: FRONT (1) | Format: (2400x1080 photo / 1920x864@60 video @ 60fps)
2023-11-08 20:24:16.659 31765-31765 PreviewView             com.mrousavy.camera.example          I  Resizing PreviewView to 1920 x 864...
2023-11-08 20:24:16.660 31765-31765 CameraView              com.mrousavy.camera.example          I  Updating CameraSession...
2023-11-08 20:24:16.660 31765-31860 CameraSession           com.mrousavy.camera.example          I  Updating CameraSession Configuration...
2023-11-08 20:24:16.661 31765-31860 CameraSession           com.mrousavy.camera.example          I  Configuring Camera #1...
2023-11-08 20:24:16.745 31765-32005 CameraManagerGlobal     com.mrousavy.camera.example          I  Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_IDLE for client com.mrousavy.camera.example API Level 2User Id 0
2023-11-08 20:24:16.860 31765-32005 CameraManagerGlobal     com.mrousavy.camera.example          I  postSingleUpdate device: camera id 0 status STATUS_PRESENT
2023-11-08 20:24:16.860 31765-31831 CameraDevices           com.mrousavy.camera.example          I  Camera #0: Available!
2023-11-08 20:24:16.860 31765-32005 CameraManagerGlobal     com.mrousavy.camera.example          I  Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client com.mrousavy.camera.example API Level 2User Id 0
2023-11-08 20:24:16.861 31765-31778 .camera.example         com.mrousavy.camera.example          W  Long monitor contention with owner mrousavy/VisionCamera.main (31860) at void android.hardware.camera2.impl.CameraDeviceImpl.close()(CameraDeviceImpl.java:1463) waiters=0 in void android.hardware.camera2.impl.CameraDeviceImpl.onDeviceError(int, android.hardware.camera2.impl.CaptureResultExtras) for 124ms
2023-11-08 20:24:16.861 31765-31860 CameraManager           com.mrousavy.camera.example          I  Camera 1: Opening...
2023-11-08 20:24:16.875 31765-32122 CameraManagerGlobal     com.mrousavy.camera.example          I  postSingleUpdate device: camera id 1 status STATUS_NOT_AVAILABLE
2023-11-08 20:24:16.875 31765-31831 CameraDevices           com.mrousavy.camera.example          I  Camera #1: Unavailable!
2023-11-08 20:24:16.883 31765-32122 CameraManagerGlobal     com.mrousavy.camera.example          I  Camera 1 facing CAMERA_FACING_FRONT state now CAMERA_STATE_OPEN for client com.mrousavy.camera.example API Level 2User Id 0
2023-11-08 20:24:16.883 31765-31860 CameraManager           com.mrousavy.camera.example          I  Camera 1: Opened!
2023-11-08 20:24:16.887 31765-31860 PreviewView             com.mrousavy.camera.example          I  Resizing PreviewView to 1920 x 864...
2023-11-08 20:24:16.887 31765-31860 CameraSession           com.mrousavy.camera.example          I  Successfully configured Camera #1!
2023-11-08 20:24:16.891 31765-31860 CameraSession           com.mrousavy.camera.example          I  Configuring Session for Camera #1...
2023-11-08 20:24:16.892 31765-31860 SurfaceOutput           com.mrousavy.camera.example          I  Closing 2400x1080 PHOTO ImageReader..
2023-11-08 20:24:16.892 31765-31860 SurfaceOutput           com.mrousavy.camera.example          I  Closing 2400x1080 Video Pipeline..
2023-11-08 20:24:16.893 31765-31860 CameraSession           com.mrousavy.camera.example          I  Adding 2400 x 1080 Photo Output in Format #256...
2023-11-08 20:24:16.894 31765-31860 CameraSession           com.mrousavy.camera.example          I  Adding 1920 x 864 Video Output in Format #34...
2023-11-08 20:24:16.894 31765-31860 VideoPipeline           com.mrousavy.camera.example          I  Initializing 1920 x 864 Video Pipeline (format: NATIVE)
2023-11-08 20:24:16.894 31765-31860 VideoPipeline           com.mrousavy.camera.example          I  Using ImageReader round-trip (format: #34)
2023-11-08 20:24:16.894 31765-31860 VideoPipeline           com.mrousavy.camera.example          I  Using API 29 for GPU ImageReader...
2023-11-08 20:24:16.895 31765-31860 CameraSession           com.mrousavy.camera.example          I  Adding 1920 x 864 Preview Output...
2023-11-08 20:24:16.895 31765-31860 PreviewView             com.mrousavy.camera.example          I  Resizing PreviewView to 1920 x 864...
2023-11-08 20:24:16.898 31765-31860 CreateCaptureSession    com.mrousavy.camera.example          I  Camera 1: Creating Capture Session #1002... Hardware Level: 0} | Outputs: [android.hardware.camera2.params.OutputConfiguration@70575b5d, android.hardware.camera2.params.OutputConfiguration@fc2c2196, android.hardware.camera2.params.OutputConfiguration@8def04b6]
2023-11-08 20:24:16.898 31765-31860 CreateCaptureSession    com.mrousavy.camera.example          I  Using new API (>=28)
2023-11-08 20:24:16.994 31765-31860 CreateCaptureSession    com.mrousavy.camera.example          I  Camera 0: Capture Session #1001 closed!
2023-11-08 20:24:16.994 31765-31860 CreateCaptureSession    com.mrousavy.camera.example          I  Camera 1: Capture Session #1002 configured!
2023-11-08 20:24:16.994 31765-31860 CameraSession           com.mrousavy.camera.example          I  Successfully configured Session with 3 outputs for Camera #1!
2023-11-08 20:24:16.994 31765-31860 CameraSession           com.mrousavy.camera.example          I  Updating Video Outputs...
2023-11-08 20:24:16.994 31765-31860 VideoPipeline           com.mrousavy.camera.example          I  Setting 1920 x 864 FrameProcessor Output...
2023-11-08 20:24:16.994 31765-31860 VideoPipeline           com.mrousavy.camera.example          I  Removing RecordingSession Output...
2023-11-08 20:24:16.996 31765-31860 CameraSession           com.mrousavy.camera.example          I  Successfully updated CameraSession Configuration! isActive: true
2023-11-08 20:24:16.996 31765-31860 CameraView              com.mrousavy.camera.example          I  invokeOnInitialized()
2023-11-08 20:24:16.997 31765-32122 CameraManagerGlobal     com.mrousavy.camera.example          I  Camera 1 facing CAMERA_FACING_FRONT state now CAMERA_STATE_ACTIVE for client com.mrousavy.camera.example API Level 2User Id 0
2023-11-08 20:24:16.997 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  Camera initialized!
2023-11-08 20:24:17.323 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.324 31765-31861 Frame                   com.mrousavy.camera.example          I  Buffer 1920 x 864 @ 1920
2023-11-08 20:24:17.361 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.427 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.464 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.496 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.535 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.563 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.598 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.640 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.664 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.712 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.733 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.774 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.800 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.839 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.868 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.913 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.938 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:17.978 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.011 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.035 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.067 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.106 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.136 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.170 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.209 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.250 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.278 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.303 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.341 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.342 31765-31861 Frame                   com.mrousavy.camera.example          I  Buffer 1920 x 864 @ 1920
2023-11-08 20:24:18.378 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.409 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.432 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.444 31765-31765 ViewRootIm...nActivity] com.mrousavy.camera.example          I  ViewPostIme pointer 0
2023-11-08 20:24:18.458 31765-31830 ReactNativeJS           com.mrousavy.camera.example          D  state: BEGAN
2023-11-08 20:24:18.468 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.498 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.526 31765-31765 ViewRootIm...nActivity] com.mrousavy.camera.example          I  ViewPostIme pointer 1
2023-11-08 20:24:18.531 31765-31765 unknown:ReactNative     com.mrousavy.camera.example          W  Can't cancel already finished gesture. Is a child View trying to start a gesture from an UP/CANCEL event?
2023-11-08 20:24:18.532 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.548 31765-31830 ReactNativeJS           com.mrousavy.camera.example          D  state: ACTIVE
2023-11-08 20:24:18.550 31765-31830 ReactNativeJS           com.mrousavy.camera.example          D  state: END
2023-11-08 20:24:18.550 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  Taking photo...
2023-11-08 20:24:18.556 31765-31765 CameraView              com.mrousavy.camera.example          D  Finding view 15...
2023-11-08 20:24:18.556 31765-31765 CameraView              com.mrousavy.camera.example          D  Found view 15!
2023-11-08 20:24:18.557 31765-32033 CameraView.takePhoto    com.mrousavy.camera.example          I  Taking photo... Options: {qualityPrioritization=speed, photoCodec=jpeg, quality=90.0, flash=off, enableShutterSound=false}
2023-11-08 20:24:18.557 31765-32033 CameraSession           com.mrousavy.camera.example          I  Photo capture 0/3 - preparing capture request (2400x1080)...
2023-11-08 20:24:18.562 31765-32033 CameraSession           com.mrousavy.camera.example          I  Photo capture 1/3 - starting capture...
2023-11-08 20:24:18.563 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.589 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.621 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.655 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.697 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.727 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.754 31765-31860 CameraSession           com.mrousavy.camera.example          I  Photo Captured!
2023-11-08 20:24:18.755 31765-31860 CameraSession           com.mrousavy.camera.example          I  Photo captured! 2400 x 1080
2023-11-08 20:24:18.756 31765-32033 CameraSession           com.mrousavy.camera.example          I  Photo capture 2/3 complete - received metadata with timestamp 205459031779501
2023-11-08 20:24:18.756 31765-32033 CameraSession           com.mrousavy.camera.example          I  Photo capture 3/3 complete - received 2400 x 1080 image.
2023-11-08 20:24:18.757 31765-32033 CameraView.takePhoto    com.mrousavy.camera.example          I  Successfully captured 2400 x 1080 photo!
2023-11-08 20:24:18.795 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.807 31765-31774 .camera.example         com.mrousavy.camera.example          W  Cleared Reference was only reachable from finalizer (only reported once)
2023-11-08 20:24:18.811 31765-31776 System                  com.mrousavy.camera.example          W  A resource failed to call Surface.release. 
2023-11-08 20:24:18.812 31765-31776 System                  com.mrousavy.camera.example          W  A resource failed to call HardwareBuffer.close. 
2023-11-08 20:24:18.812 31765-31776 libc                    com.mrousavy.camera.example          A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 31776 (FinalizerDaemon), pid 31765 (.camera.example)
2023-11-08 20:24:18.836 31765-32033 CameraView.takePhoto    com.mrousavy.camera.example          I  Successfully saved photo to file! /data/user/0/com.mrousavy.camera.example/cache/mrousavy3036235101182437960.jpg
2023-11-08 20:24:18.837 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  Media captured! {"isMirrored":true,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy3036235101182437960.jpg","isRawPhoto":false,"height":1080,"orientation":"landscape-left","width":2400}
2023-11-08 20:24:18.858 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.921 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.927 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.935 31765-31765 CameraView              com.mrousavy.camera.example          I  Updating CameraSession...
2023-11-08 20:24:18.936 31765-31860 CameraSession           com.mrousavy.camera.example          I  Updating CameraSession Configuration...
2023-11-08 20:24:18.936 31765-31860 CameraSession           com.mrousavy.camera.example          I  Successfully updated CameraSession Configuration! isActive: false
2023-11-08 20:24:18.959 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:18.995 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:19.030 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:19.063 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  Image loaded. Size: 1080x2256
2023-11-08 20:24:19.064 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:19.064 31765-31830 ReactNativeJS           com.mrousavy.camera.example          I  media has loaded.
2023-11-08 20:24:19.094 31765-31861 VideoPipeline           com.mrousavy.camera.example          I  ImageReader::onImageAvailable!
2023-11-08 20:24:19.099 31765-31780 CameraManagerGlobal     com.mrousavy.camera.example          I  Camera 1 facing CAMERA_FACING_FRONT state now CAMERA_STATE_IDLE for client com.mrousavy.camera.example API Level 2User Id 0
---------------------------- PROCESS STARTED (32251) for package com.mrousavy.camera.example ----------------------------
2023-11-08 20:24:19.363 32249-32249 DEBUG                   pid-32249                            A  Cmdline: com.mrousavy.camera.example
2023-11-08 20:24:19.363 32249-32249 DEBUG                   pid-32249                            A  pid: 31765, tid: 31776, name: FinalizerDaemon  >>> com.mrousavy.camera.example <<<
---------------------------- PROCESS ENDED (31765) for package com.mrousavy.camera.example ----------------------------
2023-11-08 20:24:19.520  1391-1458  WindowManager           system_server                        E  win=Window{6e5f636 u0 com.mrousavy.camera.example/com.mrousavy.camera.example.MainActivity EXITING} destroySurfaces: appStopped=false cleanupOnResume=false win.mWindowRemovalAllowed=true win.mRemoveOnExit=true win.mViewVisibility=0 caller=com.android.server.wm.ActivityRecord.destroySurfaces:6539 com.android.server.wm.ActivityRecord.destroySurfaces:6520 com.android.server.wm.WindowState.onExitAnimationDone:5987 com.android.server.wm.ActivityRecord$$ExternalSyntheticLambda10.accept:2 java.util.ArrayList.forEach:1262 com.android.server.wm.ActivityRecord.onAnimationFinished:8607 com.android.server.wm.ActivityRecord.postApplyAnimation:6252 
---------------------------- PROCESS ENDED (32251) for package com.mrousavy.camera.example ----------------------------

Camera Device

{
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "limited",
  "maxZoom": 4,
  "minZoom": 1,
  "supportsLowLightBoost": true,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "name": "FRONT (1)",
  "hasFlash": false,
  "hasTorch": false,
  "position": "front",
  "id": "1"
}

Device

Galaxy S21

VisionCamera Version

3.6.4

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

xulihang commented 10 months ago

Same crash with vision camera v2 + reanimated v3.

Vision camera 2.15 + reanimated v2 works okay.

LuongTruong commented 10 months ago

I got the same issue when using the vision camera v2 + reanimated v3. The reason I can't use vision camera v3 is that it support Android from 26 and above, which is only covert 93.7%

Screenshot 2023-11-20 at 10 45 17

I love the vision camera v3 a lot but can't use it at the moment. If anyone know how to make the vision camera v2 work with reanimated v3, please let me know. Many thanks!

xulihang commented 10 months ago

The latest version of vision camera v2 can work with reanimated v3: https://github.com/mrousavy/react-native-vision-camera/pull/1759

But takePhoto + frame processor seems to cause a crash.

LuongTruong commented 10 months ago

Hello guys, I am able to build the camera v2 + reanimated v3. Below is the details of my package.json

"react": "18.2.0",
"react-native": "0.72.7",
"react-native-reanimated": "3.5.4",
"react-native-vision-camera": "2.16.5",
"vision-camera-code-scanner": "0.2.0"

Below is my script to build aab and apk:

"android-aab": "cd android && ./gradlew clean && rm -rf .gradle && ./gradlew bundleRelease && cd ../",
"android-apk": "cd android && ./gradlew clean && rm -rf .gradle && ./gradlew assembleRelease && cd ../",

Hope these can help people who have the same issue as me

mrousavy commented 8 months ago

Hey does this still crash? I made some changes here and there. Also error reporting should be better now.

mrousavy commented 8 months ago

Maybe I just fixed this in https://github.com/mrousavy/react-native-vision-camera/pull/2408 - could you test and verify?

bglgwyng commented 8 months ago

I verified this fix worked.