🐛 [Android] `takePhoto` is slow

jkaufman commented 3 weeks ago

What's happening?

takePhoto is slow. A minimal app executes takePhoto in ~1.40s on a Motorola Moto G Stylus 5G, 0.66s on Samsung Galaxy S21 5G, and 0.48s on a Google Pixel 3a. After performing extensive logging, we can see significant durations separating shutter button press, capture request creation, capture request fulfillment, and image save.

Reproduceable Code

A minimal reproduction case is available at rnvc-playround/android-slow-capture-basic. Note the patch to RNVC that adds additional logging.

Relevant log output

Logs and summary statistics drawn from adb logcat, adb shell cmd media.camera watch, and application console may be found in rnvc-playground/android-slow-capture-basic/results. We reproduced the test on three different Android devices Google Pixel 3a, Motorola Moto G Stylus 5G, and Samsung Galaxy S21 5G.

Camera Device

The camera device is different across each of the Android handsets tested. The application logs at results/app-output.txt include the device name and selected format. Copied here:

Moto G

        Device name: 0 (BACK) androidx.camera.camera2
        Format: {
      "videoStabilizationModes": [
        "off",
        "cinematic"
      ],
      "autoFocusSystem": "contrast-detection",
      "photoWidth": 4080,
      "supportsPhotoHdr": false,
      "supportsDepthCapture": false,
      "maxISO": 6400,
      "minISO": 100,
      "minFps": 10,
      "videoWidth": 1920,
      "supportsVideoHdr": false,
      "videoHeight": 1080,
      "fieldOfView": 75.04000877522147,
      "maxFps": 30,
      "photoHeight": 2296
    }

S21

        Device name: 0 (BACK) androidx.camera.camera2
        Format: {
      "videoStabilizationModes": [
        "off"
      ],
      "autoFocusSystem": "contrast-detection",
      "photoWidth": 4032,
      "supportsPhotoHdr": true,
      "supportsDepthCapture": false,
      "maxISO": 3200,
      "minISO": 50,
      "minFps": 7,
      "videoWidth": 3840,
      "supportsVideoHdr": false,
      "videoHeight": 2160,
      "fieldOfView": 80.0605180983776,
      "maxFps": 30,
      "photoHeight": 3024
    }

Pixel 3a

        Device name: 0 (BACK) androidx.camera.camera2
        Format: {
      "videoStabilizationModes": [
        "off",
        "cinematic"
      ],
      "autoFocusSystem": "contrast-detection",
      "photoWidth": 4032,
      "supportsPhotoHdr": false,
      "supportsDepthCapture": false,
      "maxISO": 7111,
      "minISO": 55,
      "minFps": 7,
      "videoWidth": 3840,
      "supportsVideoHdr": false,
      "videoHeight": 2160,
      "fieldOfView": 76.94092946489485,
      "maxFps": 30,
      "photoHeight": 3024
    }

Device

Google Pixel 3a, Motorola Moto G Stylus 5G, and Samsung Galaxy S21 5G

VisionCamera Version

4.5.1

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

[X] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

maintenance-hans[bot] commented 3 weeks ago

Guten Tag, Hans here! 🍻 Thank you for providing detailed information about your issue with takePhoto. It looks like you have done some good troubleshooting already!

To further assist mrousavy with diagnosing ze problem, it would be helpful to have more context, especially any logs that you might have gathered from adb logcat during ze photo capture process. This will help understand why ze delays are occurring on different devices.

If you could include this log output, or double-check that it is sufficiently detailed, that would greatly help in resolving this issue.

Feel free to also explore sponsoring mrousavy if you find ze project useful, it helps keep everything running smoothly!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

mrousavy commented 3 weeks ago

Interesting. I guess we'd have to compare with a bare CameraX android app to see if VisionCamera does anything wrong?

mrousavy commented 3 weeks ago

0.66s on a Samsung

I mean I wouldn't be surprised if this just takes really long on Android. Android is known for slower capture than iOS. 0.66s is pretty slow, so maybe this is under the highest quality and longest exposure options available, maybe some stabilization, quality prioritization or lower format resolution speeds this up.

AjaxVM commented 3 weeks ago

We have tested this with a variety of configurations, and this is with flash off - if we lower quality to 720p, or set the balance to speed, it does speed up a bit, but is still very slow. Note that capturing logcat logs for the base camera is about 180ms on the slow end for full resolution capture, so this is much slower. It appears to be largely a delay between takePhoto starting and the callback executing for onCaptureStarted as a large part of the culprit, with the second delay (from onCaptureStarted to onImageSaved) being more related to resolution/settings.

jkaufman commented 3 weeks ago

@mrousavy Definitely worth taking a peek at the linked minimal repro case, summary results (brief.txt) for each device, and raw logs. The delay from shutter press to capture start exceeds the entire capture duration in the native camera app on the S21. there may be an opportunity to improve performance with careful CameraX session management within RNVC.

mrousavy commented 3 weeks ago

The native Samsung Camera app does not use CameraX.

If you remember, VisionCamera V3 didn't use CameraX either - it was Camera2 and it was much faster. Like, a lot faster. But unfortunately it broke for like 20% of devices and I couldn't fix issues or quirks fast enough, so CameraX it is - but I still think CameraX is fast enough - maybe there's something wrong configured, or it's a new bug/regression in CameraX.

OnCaptureStarted is definitely the first event after precapture, and precapture does AF/AE/AWB triggers which are usually the parts that are really slow. If AF/AE/AWB are already locked/focused/converged, it should be fast.

jkaufman commented 3 weeks ago

Thanks for digging in. It's unsustainable to base a device-agnostic Android camera library on Camera2. No question that Samsung can achieve faster capture on a relatively small set of supported devices using Camera2, directly. I appreciate the move to CameraX and know it offers both greater reliability and more consistent quality.

The intended purpose of this ticket is to kickstart a conversation about what RNVC might do to improve on observed CameraX performance. Perhaps there's some sort of clever CameraX session management or AF/AE/AWB defaults that might help. Or maybe this is surfacing a bug in CameraX. In any case, we're finding that a 300ms delay between shutter tap and capture start exceeds what users expect, resulting in blurry images as they move on to framing their next photo.

jkaufman commented 3 weeks ago

I ran a quick test using the Android project sample CameraXBasic. This is an outdated project running CameraX 1.1.0-beta01, so take these results with a grain of salt. I captured and saved five consecutive photos with image analysis disabled in an average duration of 306.2ms.

Tests were performed on the same Pixel 3a as above.

jkaufman commented 3 weeks ago

I repeated the test with camerax-demo, a modern sample project with more extensive use of CameraX APIs. The project was lightly modified to use the same CameraX version as the initial report, 1.4.0-beta02, as well as log timings. I captured and saved five consecutive photos in an average duration of 409.8ms. Below is abbreviated logcat output, including the timing of each onCaptureStarted and onImageSaved callback (tag CAPTURE-TIMING).

In summary, the time to onCaptureStarted is ~~130ms~~ (edit: see below) longer, on average, under RNVC than in this sample project (using Pixel 3a). I'll need to dig in to the CameraX 3A settings to see if that accounts for the difference. We'll also want to repeat these tests on the Samsung device since it exhibits the worst performance.

08-20 22:07:07.051 28990 28990 E CAPTURE-TIMING: onCaptureStarted: after 214ms
08-20 22:07:07.321 28990 28990 E CAPTURE-TIMING: onImageSaved: after 484ms
08-20 22:07:08.449 28990 28990 E CAPTURE-TIMING: onCaptureStarted: after 206ms
08-20 22:07:08.641 28990 28990 E CAPTURE-TIMING: onImageSaved: after 398ms
08-20 22:07:09.682 28990 28990 E CAPTURE-TIMING: onCaptureStarted: after 200ms
08-20 22:07:09.864 28990 28990 E CAPTURE-TIMING: onImageSaved: after 382ms
08-20 22:07:11.381 28990 28990 E CAPTURE-TIMING: onCaptureStarted: after 210ms
08-20 22:07:11.602 28990 28990 E CAPTURE-TIMING: onImageSaved: after 431ms
08-20 22:07:12.848 28990 28990 E CAPTURE-TIMING: onCaptureStarted: after 198ms
08-20 22:07:13.004 28990 28990 E CAPTURE-TIMING: onImageSaved: after 354ms

EDIT: The session was configured with CAPTURE_MODE_MAXIMIZE_QUALITY rather than RNVC's default, CAPTURE_MODE_MINIMIZE_LATENCY. With the capture mode updated, I'm observing the following timing:

2024-08-21 08:57:03.500 31636-31636 CAPTURE-TIMING E  onCaptureStarted: after 166ms
2024-08-21 08:57:03.654 31636-31636 CAPTURE-TIMING E  onImageSaved: after 320ms
2024-08-21 08:57:05.000 31636-31636 CAPTURE-TIMING E  onCaptureStarted: after 169ms
2024-08-21 08:57:05.164 31636-31636 CAPTURE-TIMING E  onImageSaved: after 333ms
2024-08-21 08:57:06.266 31636-31636 CAPTURE-TIMING E  onCaptureStarted: after 159ms
2024-08-21 08:57:06.427 31636-31636 CAPTURE-TIMING E  onImageSaved: after 319ms
2024-08-21 08:57:07.399 31636-31636 CAPTURE-TIMING E  onCaptureStarted: after 142ms
2024-08-21 08:57:07.564 31636-31636 CAPTURE-TIMING E  onImageSaved: after 308ms
2024-08-21 08:57:08.464 31636-31636 CAPTURE-TIMING E  onCaptureStarted: after 136ms
2024-08-21 08:57:08.611 31636-31636 CAPTURE-TIMING E  onImageSaved: after 283ms

This spends an average 154ms between shutter press and onCaptureStarted, which is ~181ms faster than RNVC.

AjaxVM commented 3 weeks ago

Using the sample app @jkaufman shared (capture-timing branch, with the change to use MINIMIZE_LATENCY), I came up with these timings on the Samsung S21 device from before. NOTE: the RNVC timings are the averages for that device. I am in the same location/quality of light, and the output images are the same dimensions.

Timings RNVC: onCaptureStarted: after 451ms onImageSaved: after 650ms (199ms)

camerax-demo: onCaptureStarted: after 287ms onImageSaved: 777ms (490ms)

This is closer to what I would anticipate, and is 164ms faster to the onCaptureStarted callback (though interestingly, it is slower to save the image, but that is possibly fine.)

mrousavy / react-native-vision-camera