🐛 Switching between aspect ratios

ppanwin10 commented 2 months ago

What's happening?

Hi,

When transitioning between various aspect ratios, I saw a brief pause for recalculating size and color.

Is there a way to improve it?

Record from Example app. https://github.com/mrousavy/react-native-vision-camera/assets/77748079/ddfcfe32-d717-4736-8f1d-a9b371a78eb8

This is more visible in my own app https://github.com/mrousavy/react-native-vision-camera/assets/77748079/65dd84c7-a321-44fc-883d-f239654f587d

Reproduceable Code

enum CameraRatio {
  Square = 1,
  _4_3 = 4 / 3,
  _16_9 = 16 / 9,
}

const [screenAspectRatio, setScreenAspectRatio] = useState(CameraRatio._16_9)
const format = useCameraFormat(device, [
    { fps: targetFps },
    { videoAspectRatio: screenAspectRatio },
    { videoResolution: 'max' },
    { photoAspectRatio: screenAspectRatio },
    { photoResolution: 'max' },
])

<View style={styles.rightButtonRow}>
...
<PressableOpacity
  style={styles.button}
  onPress={() => {
    switch (screenAspectRatio) {
      case CameraRatio.Square:
        setScreenAspectRatio(CameraRatio._4_3)
        break
      case CameraRatio._4_3:
        setScreenAspectRatio(CameraRatio._16_9)
        break
      case CameraRatio._16_9:
        setScreenAspectRatio(CameraRatio.Square)
        break
    }
  }}>
  <Text style={styles.text}>{`Ratio`}</Text>
</PressableOpacity>

...
</View>

<ReanimatedCamera
...
resizeMode={'contain'} 
/>

Relevant log output


12:43:18.027: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!
12:43:18.054: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
12:43:18.055: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
12:43:18.055: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
12:43:18.189: [info] 📸 VisionCamera.onCameraStopped(): Camera stopped!

WARNING: Logging before InitGoogleLogging() is written to STDERR
W0507 12:43:20.159471 1842311168 JSIExecutor.cpp:378] Memory warning (pressure level: 1) received by JS VM, unrecognized pressure level
[I] <libMMKV.mm:301::-[MMKV onMemoryWarning]> cleaning on memory warning mmkv.default
[I] <MMKV.cpp:308::clearMemoryCache> clearMemoryCache [mmkv.default]
[I] <MemoryFile.cpp:104::close> closing fd[0xb], /var/mobile/Containers/Data/Application/40EC30E7-4D3C-461E-AEEA-F1EC66872370/Documents/mmkv/mmkv.default
{
  "formats": [],
  "hardwareLevel": "full",
  "position": "back",
  "hasTorch": true,
  "minFocusDistance": 12,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minZoom": 1,
  "sensorOrientation": "landscape-right",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "isMultiCam": false,
  "hasFlash": true,
  "name": "Back Camera",
  "minExposure": -8,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 16
}
12:44:54.571: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [enableLocation]
12:44:54.571: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
12:44:54.571: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
{
  "formats": [],
  "hardwareLevel": "full",
  "position": "back",
  "hasTorch": true,
  "minFocusDistance": 12,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minZoom": 1,
  "sensorOrientation": "landscape-right",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "isMultiCam": false,
  "hasFlash": true,
  "name": "Back Camera",
  "minExposure": -8,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 16
}
[I] <MMKV_OSX.cpp:99::setIsInBackground> g_isInBackground:0
[I] <libMMKV.mm:313::+[MMKV didBecomeActive]> isInBackground:0
{
  "formats": [],
  "hardwareLevel": "full",
  "position": "back",
  "hasTorch": true,
  "minFocusDistance": 12,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "minZoom": 1,
  "sensorOrientation": "landscape-right",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "isMultiCam": false,
  "hasFlash": true,
  "name": "Back Camera",
  "minExposure": -8,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 16
}
12:44:54.979: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: true)
12:44:55.396: [info] 📸 VisionCamera.onCameraStarted(): Camera started!
12:44:55.402: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
1678585303.721708: 2160x3840 yuv Frame (portrait)
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301dfcb20>(
another test,
5
)
) (Optional(<__NSArrayI 0x301dfcb20>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
12:44:55.436: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301dfe1c0>(
another test,
5
)
) (Optional(<__NSArrayI 0x301dfe1c0>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
1678585437.21225: 2160x3840 yuv Frame (portrait)
1678585553.872458: 2160x3840 yuv Frame (portrait)
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de5d60>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de5d60>(
another test,
5
)
))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678585670.532583: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301dfe4e0>(
another test,
5
)
) (Optional(<__NSArrayI 0x301dfe4e0>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
1678585770.527041: 2160x3840 yuv Frame (portrait)
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de6620>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de6620>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678585887.187083: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de3360>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de3360>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678585987.181458: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de3700>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de3700>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678586103.841333: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301dfeec0>(
another test,
5
)
) (Optional(<__NSArrayI 0x301dfeec0>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678586220.501708: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de39c0>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de39c0>(
another test,
5
)
))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
1678586320.496041: 2160x3840 yuv Frame (portrait)
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301dff700>(
another test,
5
)
) (Optional(<__NSArrayI 0x301dff700>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExamplePlugin: 2160 x 3840 Image. Logging 5 parameters:
ExamplePlugin:   -> someObject (NSString)
ExamplePlugin:   -> someArray (NSString)
ExamplePlugin:   -> someString (NSString)
ExamplePlugin:   -> someNumber (NSString)
ExamplePlugin:   -> someBoolean (NSString)
ExampleSwiftPlugin: 2160 x 3840 Image. Logging 5 parameters:
ExampleSwiftPlugin:   -> Optional(hello!) (Optional(hello!))
ExampleSwiftPlugin:   -> Optional({
    second = test;
    test = 0;
}) (Optional({
    second = test;
    test = 0;
}))
ExampleSwiftPlugin:   -> Optional(1) (Optional(1))
ExampleSwiftPlugin:   -> Optional(<__NSArrayI 0x301de3e00>(
another test,
5
)
) (Optional(<__NSArrayI 0x301de3e00>(
another test,
5
)
))
ExampleSwiftPlugin:   -> Optional(42) (Optional(42))

Camera Device

{
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 16,
  "minFocusDistance": 12,
  "minZoom": 1,
  "hardwareLevel": "full",
  "position": "back",
  "hasTorch": true,
  "minExposure": -8,
  "hasFlash": true,
  "name": "Back Camera",
  "sensorOrientation": "landscape-right",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "formats": [],
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "neutralZoom": 1,
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false
}

Device

iPhont 11 IOS17.4.1

VisionCamera Version

4.0.3

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

mrousavy commented 2 months ago

Yea - this is expected - you are updating the Format on a Camera, this just needs to be reconfigured. You can trick this by cutting the view yourself, and then cropping the photo in post processing.

Also, your logs don't include the part where the format gets switched, so I can't really say if there is a way to improve this or not.

I'm specifically looking for these logs:

12:44:54.979: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: true)

It should say 1 props: [format]. If that's the case, then it's as fast as it can get.

ppanwin10 commented 2 months ago

Yea - this is expected - you are updating the Format on a Camera, this just needs to be reconfigured. You can trick this by cutting the view yourself, and then cropping the photo in post processing.

Also, your logs don't include the part where the format gets switched, so I can't really say if there is a way to improve this or not.

I'm specifically looking for these logs:
12:44:54.979: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
12:44:54.979: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: true)
It should say 1 props: [format]. If that's the case, then it's as fast as it can get.

ok thank you, let me go back to see my log. Cutting the view manually sounds a good idea!

mrousavy / react-native-vision-camera