🐛 The resolution of the PhotoFile is different than the format photo resolution

[Dingenis]

What's happening?

When taking a photo the PhotoFile has a different resolution than the format.

Reproduceable Code

let device = useCameraDevice(cameraPosition)
const format = useCameraFormat(device, [{ photoResolution: 'max' }])

useEffect(() => {
    const f =
      format != null
        ? `(${format.photoWidth}x${format.photoHeight} photo / ${format.videoWidth}x${format.videoHeight}@${format.maxFps} video @ ${fps}fps)`
        : undefined
    console.log(`Camera: ${device?.name} | Format: ${f}`)
  }, [device?.name, format, fps])

return   
 <ReanimatedCamera
  style={StyleSheet.absoluteFill}
  device={device}
  isActive={isActive}
  ref={camera}
  format={format}
  photo={true}
  video={true}
 />

Relevant log output

Metro:

 LOG  Camera: 1 (FRONT) androidx.camera.camera2 | Format: (3264x2448 photo / 1920x1080@30 video @ 30fps)
 LOG  Media captured! {"isMirrored":true,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-5497026049884411527.jpg","isRawPhoto":false,"height":1080,"orientation":"portrait","width":1440}
 LOG  Image loaded. Size: 1072x1440

Logcat:

2024-06-27 12:55:12.082 16956-18577 CameraView.takePhoto    com.mrousavy.camera.example          I  Successfully captured 1440 x 1080 photo!

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-right",
  "hardwareLevel": "limited",
  "maxZoom": 4,
  "minZoom": 1,
  "maxExposure": 20,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 0,
  "minExposure": -20,
  "name": "1 (FRONT) androidx.camera.camera2",
  "hasFlash": false,
  "hasTorch": false,
  "position": "front",
  "id": "1"
}

Device

Samsung Galaxy A53 (Android 14)

VisionCamera Version

4.3.1

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• [ ] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[maintenance-hans[bot]]

Guten Tag, Hans here.

[!NOTE] New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time. To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖. Sponsored issues will be prioritized.

[Dingenis]

Hi @mrousavy, our company (Spaces Experiences) has been a sponsor for around four months and we are very happy that this library exists, so thank you! This issue I posted here has priority for us, if you have any time taking a look would be much appreciated. Cheers, Dingenis

[mrousavy]

Hi! Thanks for the sponsorship - does this also happen if you disable video?

[Dingenis]

Hi, thanks for your quick reply. Yes, with only video={false} it still happens. However with video={false} preview={false} the photo does have the same resolution (as the format):

 LOG  Media captured! {"isMirrored":true,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-7410530320673441399.jpg","isRawPhoto":false,"height":2448,"orientation":"portrait","width":3264}

[JustinHaut]

fwiw visionCamera version 4.3.2 with my Galaxy S22 doesn't have this issue. I tried disabling/enabling video and preview, and verified both front and back camera format and photo resolutions match.

[Dingenis]

Yes good point @JustinHaut, I also tested on a Galaxy A41 and there it does indeed also match.

[mrousavy]

Okay thanks, good research on preview={false} that is actually quite helpful - I just merged a PR a few days ago that changes the Preview resolution selector to now use the format's video resolution, just like on iOS. (https://github.com/mrousavy/react-native-vision-camera/pull/3026)

This might fix this issue.

[Dingenis]

@mrousavy I can reproduce it with the example app with video={false} on the following devices (via Android Studio Device Streaming):

Pixel Fold

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4000x3000 photo / 3840x2160@60 video @ 60fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-2125636817995574257.jpg","isRawPhoto":false,"height":2160,"orientation":"portrait","width":3840}

Pixel 8 Pro

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4000x2000 photo / 3840x2160@60 video @ 60fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-7321524609184097620.jpg","isRawPhoto":false,"height":2160,"orientation":"landscape-right","width":3840}

Pixel 7a

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4000x2000 photo / 3840x2160@60 video @ 60fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-2799048483279202367.jpg","isRawPhoto":false,"height":1080,"orientation":"landscape-right","width":1920}

Samsung Galaxy A53

LOG  Camera: 1 (FRONT) androidx.camera.camera2 | Format: (3264x2448 photo / 1920x1080@30 video @ 30fps)
LOG  Media captured! {"isMirrored":true,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-5497026049884411527.jpg","isRawPhoto":false,"height":1080,"orientation":"portrait","width":1440}

And I cannot reproduce it on these devices:

Samsung Galaxy S23 Ultra

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4000x1868 photo / 3840x2160@30 video @ 30fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-9181039376298899107.jpg","isRawPhoto":false,"height":1868,"orientation":"landscape-right","width":4000}

SHARP AQUOS sense2 SH-01L

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (864x480 photo / 1920x1080@30 video @ 30fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-1548835093455550811.jpg","isRawPhoto":false,"height":480,"orientation":"landscape-right","width":864}

[Dingenis]

Okay thanks, good research on preview={false} that is actually quite helpful - I just merged a PR a few days ago that changes the Preview resolution selector to now use the format's video resolution, just like on iOS. (#3026)

This might fix this issue.

Good to hear! Let me know if can help in any way once you get to it (like testing or something). To add I used the main branch to test on the devices shown above, so I don't think (https://github.com/mrousavy/react-native-vision-camera/pull/3026) fixes the issue.

[mrousavy]

The problem here is that the Formats API on Android is not 100% accurate, which sucks - I know.

On Android, there is no API to figure out which combination of output resolutions is supported. On iOS there is, which is why the format always determines exactly what resolution the photo and video will have.

But on Android, it's always only an estimate, the OS might not be able to support 4k Photo and 4k Video at the same time, only individually. So in this case, CameraX is smart enough to fall back to a lower resolution, either for photo or for video.

I try to get an accurate list of supported video/photo resolutions here; https://github.com/mrousavy/react-native-vision-camera/blob/9ca6643748c0b39aa0d3e4fd4ebfb9c377adc0c8/package/android/src/main/java/com/mrousavy/camera/core/CameraDeviceDetails.kt#L116-L119

..but again, CameraX might choose a different resolution if it feels like it. I raised this concern with the Google/CameraX team, and after many months they came back to me and said they are going to implement an API that actually gives you accurate resolution (and FPS) combinations - which is what VisionCamera's format API is about.

So we might see this sometime soon, but for now this is all we have on Android unfortunately.

[Dingenis]

Hey, thanks for your explanation and your time! That's a bummer.... seems kind of stupid for Android to not have such a API, but I don't know too much of the Android camera world so idk 🤷 (maybe it's really hard to implement with all the different vendors who might not fully comply)

It does make it really hard to manually select a format, since the format you select can be one that is not compatible thus resulting in the photo resolution (and sometimes aspect ratio) being different than expected. When looking at the Pixel 7a it's a massive difference instead of 4000x2000 photo you get a 1920x1080 photo. So you think you are getting the highest resolution picture while instead you might have a format which actually has a lower photo resolution than maybe a other one where the video resolution is lower (which is compatible). At least if I understand it correctly? 🤔

I was thinking is there a running issue for this in the CameraX issue tracker, maybe we as community could upvote it so that they might prioritize this? Furthermore, I was wondering do you have any advice on how to deal with this? Like should formats with a high resolution and and high video ouput be avoided (because they might be invalid)?

Yet again, thanks for your time. Cheers, Dingenis.

[mrousavy]

Formats with high resolutions should not be avoided, no.

But the useCameraFormat/getAvailableCameraDevices should just be accurate in telling us which video/photo resolution combinations are supported.

[Dingenis]

Thanks for your reply @mrousavy! Hopefully, the Android team will pick this up somewhere in the near future.

I misunderstood some things on how this works and have since thought about it some more. So I have one more question, if you would indulge me, I see that the aspect ratio of the preview is set to the format's photo aspect ratio (if using photo={true}) here. Let's say the format is a "invalid" combination thus CameraX falls back on another resolution, then the aspect ratio of the preview should also change to that new aspect ratio. Currently it doesn't, is that because we only know the aspect ratio of the fallback once the user captures a picture?

[mrousavy]

This has been changed recently and now works exactly like on iOS - preview will try to match the format's video resolution and aspect ratio, with the priority being the aspect ratio.

[mrousavy]

Hey can you check if release 4.4.0 changed anything?

[Dingenis]

Hey, thanks again for you effort! I just tested it on my device, and indeed the preview confirms to the format's video resolution and aspect ratio. The issue with the resolution being lower for the ImageCapture then format still happens. Interestingly, I tried the CameraXBasic sample project (after upgrading it to the latest deps) and there the resolution is high (like the format in react-native-vision-camera).

[Dingenis]

The code from the sample project boils down to this:

val rotation = fragmentCameraBinding.viewFinder.display.rotation

// CameraProvider
val cameraProvider = cameraProvider ?: throw IllegalStateException("Camera initialization failed.")

// CameraSelector
val cameraSelector =
    CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

// Preview
val previewResolutionSelector =
    ResolutionSelector.Builder()
        .setAllowedResolutionMode(ResolutionSelector.PREFER_CAPTURE_RATE_OVER_HIGHER_RESOLUTION)
        .build()

preview =
    Preview.Builder()
        .setResolutionSelector(previewResolutionSelector)
        .setTargetRotation(rotation)
        .build()

// ImageCapture
val imageCaptureResolutionSelector =
    ResolutionSelector.Builder()
        .setAllowedResolutionMode(ResolutionSelector.PREFER_HIGHER_RESOLUTION_OVER_CAPTURE_RATE)
        .build()

imageCapture =
    ImageCapture.Builder()
        .setCaptureMode(ImageCapture.CAPTURE_MODE_MAXIMIZE_QUALITY)
        .setResolutionSelector(imageCaptureResolutionSelector)
        .setTargetRotation(rotation)
        .build()

// Must unbind the use-cases before rebinding them
cameraProvider.unbindAll()

try {
    // A variable number of use-cases can be passed here -
    // camera provides access to CameraControl & CameraInfo
    camera = cameraProvider.bindToLifecycle(this, cameraSelector, imageCapture, preview)

    val previewResolution = preview?.resolutionInfo?.resolution

    if (previewResolution != null) {
        Log.i("DingenisCamera", "${previewResolution.width}x${previewResolution.height}")
    }

    val imageCaptureResolution = imageCapture?.resolutionInfo?.resolution

    if (imageCaptureResolution != null) {
        Log.i("DingenisCamera", "${imageCaptureResolution.width}x${imageCaptureResolution.height}")
    }

    // Attach the viewfinder's surface provider to preview use case
    preview?.setSurfaceProvider(fragmentCameraBinding.viewFinder.surfaceProvider)
    observeCameraState(camera?.cameraInfo!!)
} catch (exc: Exception) {
    Log.e(TAG, "Use case binding failed", exc)
}

Which gives me the following preview and image capture resolution, this matches the highest resolution format from react-native-vision-camera (but there CameraX choses a different resolution):

2024-07-04 14:58:24.154 31948-31948 DingenisCamera                    com.android.example.cameraxbasic     I  1440x1080
2024-07-04 14:58:24.154 31948-31948 DingenisCamera                    com.android.example.cameraxbasic     I  4624x3468

[Dingenis]

Alright @mrousavy, I think I found the culprit! I was playing around with the configuration of the Preview in the CameraXBasic sample project using Preview.Builder and that's when I noticed when I used setTargetFrameRate with a min of 21 or higher the ImageCapture resolution would change to a low value (1440x1080). Whereas if 20 was used as min (setTargetFrameRate(Range(20, 30))) the higher resolution would be used (4624x3468).

In react-native-vison-camera the format says it has a maxFps of 30, but it seems that is not accurate because when changing the fps camera prop to 20 the format now does work!

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4624x3468 photo / 1920x1080@30 video @ 30fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-1755692348160762567.jpg","isRawPhoto":false,"height":3468,"orientation":"portrait","width":4624}

[mrousavy]

Oh wow interesting, great research!

I'm thinking what a good solution to this would be, maybe I'll just always allow minFps of 20...

[Dingenis]

Thanks! Good question, well to set it to a min of 20 seems like a simple solution, so i like it 😁😂 . I don't know if there are any other formats which have an even lower maxFps, so that could be a issue.

Another solution could be to change the fps camera prop to a range, in that way the responsibility is with the app maker to select a suitable range. That might be less confusing, since if you set the fps camera prop to 30 but you get 20 one might be confused. But that could also just be solved by added some documentation like: "on android the fps is target and is therefore not guaranteed" so I am not sure what is best either.

[Dingenis]

I can also confirm that lowering the fps also fixes the format on these devices (prob the rest mentioned early too):

Pixel 7a

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4624x3472 photo / 3840x2160@60 video @ 30fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-4166822674804579466.jpg","isRawPhoto":false,"height":3472,"orientation":"landscape-right","width":4624}

Pixel Fold

 LOG  Camera: 0 (BACK) androidx.camera.camera2 | Format: (4000x3000 photo / 3840x2160@60 video @ 30fps)
 LOG  Media captured! {"isMirrored":false,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy-128395477126160525.jpg","isRawPhoto":false,"height":3000,"orientation":"portrait","width":4000}