search

mrousavy / react-native-vision-camera

📸 A powerful, high-performance React Native Camera library.
https://react-native-vision-camera.com
MIT License
7.56k stars 1.1k forks source link

🐛 Frame Drops in Recorded Video When Using useFrameProcessor, #3147

Closed rmethaniya closed 2 months ago

rmethaniya commented 2 months ago

What's happening?

When the frame processor is disabled, the recorded video is smooth and free of frame drops. However, when the frame processor is enabled, the recorded video shows noticeable frame drops.

The issue occurs when using the useFrameProcessor to resize frames and play them every second with runAtTargetFps. Additionally, runAsync is used for frame processing, and frame detection is performed with react-native-fast-tflite using runSync.

Environment:

Reproduceable Code

const format = getCameraFormat(device, [
    {fps: 120}, 
    {videoResolution: 'max'}
  ])

useFrameProcessor(
  (frame: Frame) => {
    'worklet'
    const resizes = resize(frame, {
      scale: {
        width: 224,
        height: 480
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    })
    runAtTargetFps(1, () => {
      'worklet'
      runAsync(frame, () => {
        'worklet'

          const uint8 = Float32Array.from(resized, (value) => value / 255)
          const dict = model.runSync([uint8])[0]

      })
    })
  }
)

Relevant log output

Installing bindings...
Successfully installed!
Looking up Frame Processor Plugin "resize"...
Frame Processor Plugin "resize" found! Initializing...
'--122------>', '{\n  "minFocusDistance": 15,\n  "formats": [],\n  "minZoom": 1,\n  "hasTorch": true,\n  "maxExposure": 8,\n  "supportsLowLightBoost": false,\n  "sensorOrientation": "landscape-right",\n  "isMultiCam": false,\n  "hardwareLevel": "full",\n  "position": "back",\n  "hasFlash": true,\n  "minExposure": -8,\n  "name": "Back Camera",\n  "maxZoom": 123.75,\n  "physicalDevices": [\n    "wide-angle-camera"\n  ],\n  "supportsRawCapture": false,\n  "neutralZoom": 1,\n  "supportsFocus": true,\n  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0"\n}'
Loading Tensorflow Lite Model 130
Resolved Model path: http://192.168.29.212:8081/assets/src/assets/detect_int8.tflite?platform=ios&hash=d9b3a7d2cdaab8be0c2b5d9ecd44c620
Fetching http://192.168.29.212:8081/assets/src/assets/detect_int8.tflite?platform=ios&hash=d9b3a7d2cdaab8be0c2b5d9ecd44c620...
19:02:18.580: [info] 📸 VisionCamera.didSetProps(_:): Updating 28 props: [onInitialized, cameraId, position, enableBufferCompression, preview, onCodeScanned, onStarted, top, borderWidth, right, isActive, pixelFormat, video, onViewReady, onError, onStopped, overflow, enableFrameProcessor, enableZoomGesture, format, left, audio, bottom, borderRadius, zoom, photo, borderColor, onShutter]
19:02:18.580: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
19:02:18.583: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
19:02:18.583: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Input Device...
19:02:18.583: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:0...
19:02:18.587: [info] 📸 VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
19:02:18.587: [info] 📸 VisionCamera.configureOutputs(configuration:): Configuring Outputs...
19:02:18.587: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Photo output...
19:02:18.588: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Video Data output...
19:02:18.589: [info] 📸 VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
19:02:18.589: [info] 📸 VisionCamera.configureFormat(configuration:device:): Configuring Format (1920x1080 | 1920x1080@120.0 (ISO: 34.0..1360.0))...
19:02:18.590: [info] 📸 VisionCamera.configureFormat(configuration:device:): Successfully configured Format!
19:02:18.592: [info] 📸 VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA", "&8v0", "-8v0", "&8f0", "-8f0", "&BGA", "-BGA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={false})
19:02:18.592: [info] 📸 VisionCamera.getPixelFormat(for:): Using PixelFormat: 420f...
19:02:18.842: [info] 📸 VisionCamera.onCameraStarted(): Camera started!
19:02:18.842: [info] 📸 VisionCamera.onSessionInitialized(): Camera initialized!
19:02:18.842: [info] 📸 VisionCamera.configure(_:): Beginning AudioSession configuration...
19:02:18.842: [info] 📸 VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
19:02:18.844: [info] 📸 VisionCamera.configure(_:): Committed AudioSession configuration!
19:02:18.846: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
19:02:18.849: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!
Initialized TensorFlow Lite runtime.
INFO: Initialized TensorFlow Lite runtime.
19:02:19.528: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): Starting Video recording...

19:02:19.530: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): Will record to temporary file: /private/var/mobile/Containers/Data/Application/155BA7DF-5F5B-4984-A0A0-D300BFCB04AE/tmp/ReactNative/B5F80644-28A5-4EA0-A34B-55E98D5FEDAC.mp4

Model loaded!
19:02:19.538: [info] 📸 VisionCamera.recommendedVideoSettings(forOptions:): Using codec AVVideoCodecType(_rawValue: hvc1)...
19:02:19.538: [info] 📸 VisionCamera.initializeVideoWriter(withSettings:): Initializing Video AssetWriter with settings: ["AVVideoWidthKey": 1080, "AVVideoCompressionPropertiesKey": {
    AllowFrameReordering = 1;
    AllowOpenGOP = 1;
    AverageBitRate = 22498560;
    BaseLayerFrameRate = 30;
    ExpectedFrameRate = 120;
    MaxAllowedFrameQP = 41;
    MaxKeyFrameIntervalDuration = 1;
    MinAllowedFrameQP = 15;
    MinimizeMemoryUsage = 1;
    Priority = 80;
    ProfileLevel = "HEVC_Main_AutoLevel";
    RealTime = 1;
    RelaxAverageBitRateTarget = 1;
}, "AVVideoCodecKey": hvc1, "AVVideoHeightKey": 1920]
19:02:19.551: [info] 📸 VisionCamera.initializeVideoWriter(withSettings:): Initialized Video AssetWriter.
19:02:19.551: [info] 📸 VisionCamera.start(clock:): Starting Asset Writer(s)...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Allocating SharedArray (size: 8294400)...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Allocating SharedArray (size: 430080)...
Allocating _tempResizeBuffer (size: 2076928)...
Allocating SharedArray (size: 322560)...
Converting ARGB_8 Frame to RGB_8...
19:02:27.330: [info] 📸 VisionCamera.start(clock:): Asset Writer(s) started!
19:02:27.331: [info] 📸 VisionCamera.start(clock:): Started RecordingSession at time: 1114638.225520833
19:02:27.331: [info] 📸 VisionCamera.startRecording(options:onVideoRecorded:onError:): RecordingSesssion started in 7802.990458ms!
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
nw_path_necp_check_for_updates Failed to copy updated result (22)
nw_connection_copy_connected_local_endpoint_block_invoke [C34] Client called nw_connection_copy_connected_local_endpoint on unconnected nw_connection
nw_connection_copy_connected_remote_endpoint_block_invoke [C34] Client called nw_connection_copy_connected_remote_endpoint on unconnected nw_connection
nw_connection_copy_protocol_metadata_internal_block_invoke [C34] Client called nw_connection_copy_protocol_metadata_internal on unconnected nw_connection
ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...
[ -1 ], ''

ResizePlugin: Scaling to 224.000000 x 480.000000.
ResizePlugin: Rotation not specified, defaulting to: 0
ResizePlugin: Mirror: NO
ResizePlugin: Cropping to 896.000000 x 1920.000000 at (92.000000, 0.000000).
ResizePlugin: Using target format: rgb
ResizePlugin: Using target data type: uint8
Converting YUV Frame to RGB...
Resizing ARGB_8 Frame to 224.000000 x 480.000000...
Converting ARGB_8 Frame to RGB_8...

Camera Device

{
  "hardwareLevel": "full",
  "formats": [],
  "isMultiCam": false,
  "hasTorch": true,
  "position": "back",
  "neutralZoom": 1,
  "supportsFocus": true,
  "supportsRawCapture": false,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minZoom": 1,
  "minFocusDistance": 15,
  "maxZoom": 123.75,
  "sensorOrientation": "landscape-right", -> Phone Orientation is Portrait 
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "hasFlash": true,
  "name": "Back Camera",
  "minExposure": -8
}

Device

iPhone 14 (iOS 17.5.1)

VisionCamera Version

4.0.5

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

maintenance-hans[bot] commented 2 months ago

Guten Tag, Hans here! 🍻

Thanks for providing detailed information about your issue. It seems like you have done ze proper steps, but I noticed you didn't try to reproduce this issue in ze VisionCamera Example app. Please give it a try, as this can help narrow down ze problem.

Also, if you could provide some logs directly from Xcode (for iOS) or adb logcat (for Android), it would really assist mrousavy in diagnosing ze issue.

Let’s try to get this fixed!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

rmethaniya commented 2 months ago

@mrousavy I've attached the xCode logs. Please let me know if you need any more information.

mrousavy commented 2 months ago

Well, resizing a max resolution frame might just take a while. I'm actually surprised that this even works, I think resizing should be handled inside runAsync.

rmethaniya commented 2 months ago

I am encountering a crash in my application when using the resize function inside the runAsync method. The app fails to execute properly, which seems to be related to the asynchronous execution within the resize call.

`runAsync(frame, () => { 'worklet';

const resizes = resize(frame, {
  scale: {
    width: 224,
    height: 480,
  },
  pixelFormat: 'rgb',
  dataType: 'uint8',
});
const resized = Object.values(resizes);
const uint8 = Float32Array.from(resized, value => value / 255);
const dict = model.runSync([uint8])[0];

}); `

Expected, The application should process the image resizing and run the model without crashing.

mrousavy commented 2 months ago

Hm that's weird. I remember this working back when I tested it. As of right now, I don't have any free time to investigate this though.