🐛 Frame dimensions passed in to frame processor different for ios and android

titanium-cranium commented 1 month ago

What's happening?

We have a functional iOS app using RNVC 4.0.1 that utilizes frame processors to great effect. We are passing 4K images to the frame processor plugin without any issue. The CameraFormat is chosen specifically to be certain the device can handle 4K and a minimum FPS of 30 all of which is fine and when we log out the frame size in the iOS plugin, it confirms that the size is 3840x2160 (height x width).

We are now in the process of porting the app to android and recently discovered that the frame dimensions being passed to the android plugin are half the size of that being passed to the iOS plugin (1080Hx1920W). The JS code is identical for both platforms, only the plugins differ. The camera format on Android (using Samsung S23FE) shows it is set up for 2160H x 3840W but when we log out frame.height and frame.width in the plugin (without doing any other processing) it reads out 1080Hx1920W. The relevant Camera code is pasted in below.

Reproduceable Code

// Code to set the camera format:
  const format = useMemo(() => {
    if (!device || !device.formats) return { device: null, format: null }
    const formats = device.formats
    let result = formats

    // find the first format that includes the given FPS and 4K format
    const format = result.find((f) => f.maxFps >= fps && f.videoWidth == 3840)
    console.log('FORMAT', JSON.stringify(format))
    return format
  }, [device])
  return {
    device,
    format,
  }

// Camera
              <Camera
                ref={cameraRef}
                style={StyleSheet.absoluteFill}
                device={device}
                format={format}
                isActive={!video && isFocused}
                fps={fps}
                hdr
                video
                audio={microphonePermission}
                photo
                onError={console.error}
                orientation="portrait"
                focusable
                enableZoomGesture
                lowLightBoost={device.supportsLowLightBoost}
                supportsParallelVideoProcessing
                frameProcessor={frameProcessor}
                frameProcessorFps={fps}
                pixelFormat={'rgb'}
                enableFpsGraph={true}
              />

// frameProcessor code:
  const frameProcessor = useFrameProcessor(
    async (frame) => {
      'worklet'
      console.log('FRAME PROCESSOR HxW: ', frame.height, frame.width)

      try {
        const data = labelImage(frame)
      } catch (error) {
        console.log('Error in frameProcessor', error)
      }

      // const frameLabels = data ? JSON.parse(data) : []
      // updateBoundingBoxJS(frameLabels, isRecording)
    },
    [isRecording, labelConfidence]
  )

Relevant log output

'FORMAT', '{"videoStabilizationModes":["off"],"autoFocusSystem":"contrast-detection","photoWidth":4080,"supportsPhotoHdr":true,"supportsDepthCapture":false,"maxISO":3200,"minISO":50,"minFps":8,"videoWidth":3840,"supportsVideoHdr":false,"videoHeight":2160,"fieldOfView":85.2639483493727,"maxFps":30,"photoHeight":3060}'

Camera Device

{
   "formats":[

   ],
   "sensorOrientation":"landscape-left",
   "hardwareLevel":"full",
   "maxZoom":10,
   "minZoom":0.550000011920929,
   "maxExposure":20,
   "supportsLowLightBoost":true,
   "neutralZoom":1,
   "physicalDevices":[
      "wide-angle-camera",
      "ultra-wide-angle-camera",
      "wide-angle-camera",
      "telephoto-camera"
   ],
   "supportsFocus":true,
   "supportsRawCapture":false,
   "isMultiCam":true,
   "minFocusDistance":10,
   "minExposure":-20,
   "name":"0 (BACK) androidx.camera.camera2",
   "hasFlash":true,
   "hasTorch":true,
   "position":"back",
   "id":"0"
}

Device

Samsung S23FE

VisionCamera Version

4.0.1

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

maintenance-hans[bot] commented 1 month ago

Guten Tag, Hans here.

[!NOTE] New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time. To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖. Sponsored issues will be prioritized.

mrousavy commented 1 month ago

That's because frame.orientation is landscape on Android.

mrousavy commented 1 month ago

Check out Frame Orientation

titanium-cranium commented 1 month ago

@mrousavy I'm not certain I understand that solution. The format for the camera is found on the android device 3840w x 2160h (which we expected because of Android, see the log above) but the frame dimensions in the frame processor are 1920w x 1080h. Half the resolution. I modified our format retrieval to use the aspect ratio and max resolution like the example app and got the same response. I did manage to spin up the most recent main of the example app and check there and the frame processor is reporting the correct frame size. What am I missing? How would the frame dimensions change from the format being passed in to the camera to the frame being passed to the frame processor?

mrousavy commented 1 month ago

I'm sorry, what exactly is the issue?

Are you saying width and height is flipped?

titanium-cranium commented 1 month ago

@mrousavy No, not flipped, halved. I've tried lots of camera configs to request the camera return 4K video frames 3840wx2160h on android with 30fps. When I log out the format that gets passed in to the Camera, it is exactly what is expected.

'FORMAT', '{"videoStabilizationModes":["off"],"autoFocusSystem":"contrast-detection","photoWidth":4080,"supportsPhotoHdr":true,"supportsDepthCapture":false,"maxISO":3200,"minISO":50,"minFps":8,"videoWidth":3840,"supportsVideoHdr":false,"videoHeight":2160,"fieldOfView":85.2639483493727,"maxFps":30,"photoHeight":3060}'

But when I log out frame size in the frame processor itself in our app (frame.image.width or frame.image.height) I get 1920w x 1080h. What I'm trying to work out is how that could happen. When I do the same thing in the example app, it does exactly what I expect and logs out 3840wx2160h in the frame processor. So, the library is doing exactly what it is supposed to.

So, IIUC that leaves me with two potential places where something is going wrong.

1) The format being passed in to the Camera is somehow not having the desired effect and the camera is operating at 1920w x 1080h

2) The Camera is operating as specified, but the frame being sent to the frame processor is half the resolution.

I have nowhere near the insight into Camera2 or CameraX that you have, I was hoping you could point me in the right direction on where to start looking. Or suggest some debugging strategies. I've tried passing in no format. That returns a frame of 640x480. I've tried specifying only the aspect ratio of 3840/2160 and specifying 'max' video resolution as is done in the example app. But no matter what I've tried, it still logs out the same 1920wx1080h in the frame processor.

titanium-cranium commented 1 month ago

@mrousavy So after much more debugging we finally discovered that removing the photo specification lines in useCameraFormat will also alter the video resolution. It returns exactly half the requested frame resolution in the frame processor. When we added those two lines in our android app, suddenly the frame processor uses 3840x2160 again. Has no impact on ios for some reason.

Example app CameraPage.tsx. https://github.com/mrousavy/react-native-vision-camera/blob/77e98178f84824a0b1c76785469413b64dc96046/package/example/src/CameraPage.tsx#L71

  const format = useCameraFormat(device, [
    { fps: targetFps },
    { videoAspectRatio: screenAspectRatio },
    { videoResolution: 'max' },
    { photoAspectRatio: screenAspectRatio },    <-----   Remove these two lines
    { photoResolution: 'max' },                             <-----
  ])

mrousavy commented 1 month ago

Ah - yes this is pretty obvious then. Not every combination is supported, useCameraFormat tries it's best to find a good matching format but maybe it was torn between the two and it considered the higher photo one a better solution.

titanium-cranium commented 1 month ago

@mrousavy

Hi Marc:
That seems like a very misleading API though. Up until yesterday I had ignored the photoAspectRatio and photoResolution because :

1) I am exclusively using RNVC for video streaming and

2) neither argument was necessary to achieve max resolution in the frame processor on iOS

3) The CameraFormat being returned with or without specifying the photoAspectRatio and photoResolution specifies a frame size of 3840x2160 on both iOS and Android

Why should a user need to specify photoAspectRatio and photoResolution to achieve max resolution in a frame processor on android? I would politely suggest this be opened back up as a bug.

Best regards,

Brendan

mrousavy commented 1 month ago

Oh so maybe I read this wrong - you are saying when photoResolution is NOT passed to useCameraFormat(..) it will choose a lower resolution for video than when it is actually passed?

This is definitely a mistake then - if only videoResolution is set in the filter then it should find the closest matching resolution for that on the native side (frame processor and video recording).

Can you confirm that if you only pass videoResolution it will not find the desired videoResolution for you?

The more constraints you give to useCameraFormats the harder it is to meet them all. It will be ranked by order of constraint in the given list.

titanium-cranium commented 1 month ago

@mrousavy That's correct. You can prove this to yourself in the android example app by commenting out L75 & L76 and spinning it up. The FrameProcessor will log out a frame size half of max resolution. (at least on my Samsung S23 FE). https://github.com/mrousavy/react-native-vision-camera/blob/77e98178f84824a0b1c76785469413b64dc96046/package/example/src/CameraPage.tsx#L75

Before/After in the logs:

LOG  1469563135389: 3840x2160 yuv Frame (landscape-left)
 LOG  1469696405233: 3840x2160 yuv Frame (landscape-left)
 LOG  1469796418904: 3840x2160 yuv Frame (landscape-left)
 LOG  1469929740115: 3840x2160 yuv Frame (landscape-left)
 LOG  RNVSC FPS:  30
 LOG  1469996411951: 3840x2160 yuv Frame (landscape-left)
 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4080x3060 photo / 3840x2160@30 video @ 30fps)
 LOG  1470196404959: 3840x2160 yuv Frame (landscape-left)

// COMMENT OUT L75 & L76

 LOG  Camera initialized!
 LOG  RNVSC FPS:  30
 LOG  RNVSC FPS:  30
 LOG  RNVSC FPS:  30
 LOG  RNVSC FPS:  30
 LOG  RNVSC FPS:  30
 LOG  1470788600193: 1920x1080 yuv Frame (landscape-left)
 LOG  1470921934373: 1920x1080 yuv Frame (landscape-left)
 LOG  1471021962928: 1920x1080 yuv Frame (landscape-left)
 LOG  1471155272225: 1920x1080 yuv Frame (landscape-left)

mrousavy commented 1 month ago

Hm, maybe it's because I use PREFER_CAPTURE_RATE_OVER_HIGHER_RESOLUTION, and if it doesn't have a strong resolution constraint (e.g. the one from photoResolution) it will prefer a fast capture rate? https://github.com/mrousavy/react-native-vision-camera/blob/77e98178f84824a0b1c76785469413b64dc96046/package/android/src/main/java/com/mrousavy/camera/core/CameraSession%2BConfiguration.kt#L186-L193

mrousavy commented 1 month ago

Either way - I don't consider this a high priority or critical issue - the getCameraFormat APIs are just helpers. You can easily write such filters yourself if those don't fit your needs.

If you find a fix for this, PRs are of course appreciated :)

mrousavy / react-native-vision-camera