🐛 `session/cannot-create-session` Error when using Camera

elliottkember commented 1 year ago

What's happening?

CameraSessionCannotBeConfiguredError: [session/cannot-create-session] Failed to create a Camera Session for Camera 0! Outputs: [PREVIEW (1920 x 1080), PHOTO (4080 x 3072 in format #256)]

I've just updated to react-native-vision-camera v3 (thank you!) and I'm seeing a strange error, but only in AWS Device Farm. So far I have not been able to replicate this on a physical device, so I'm posting this bug report to track my progress fixing this, in case anybody else hits the same issue.

This issue may be caused by something in the upgrade to React Native 0.72, during which time I updated this library to v3 from v2.
The problem is not replicable in the emulator, which leads me to believe this is a device format mismatch
At this stage I suspect it's something with the format prop.

Reproduceable Code

const ReanimatedCamera = Reanimated.createAnimatedComponent(Camera);
Reanimated.addWhitelistedNativeProps({ zoom: true });

  const devices = keyBy(useCameraDevices(), (d) => d.position);
  const device = frontCamera ? devices.front : devices.back;
  const neutralZoom = device?.neutralZoom ?? 2;
  const minZoom = device?.minZoom ?? neutralZoom;
  const maxZoom = Math.min(device?.maxZoom ?? neutralZoom, MAX_ZOOM_FACTOR);
  const zoom = useSharedValue(neutralZoom);

<ReanimatedCamera
  enableHighQualityPhotos
  orientation={cameraOrientation}
  hdr
  photo
  video={false}
  ref={camera}
  isActive
  device={device}
  zoom={zoom.value}
  style={styles.camera}
  enableZoomGesture
  animatedProps={animatedProps}
/>

First thing to try: adding the format property to the camera component

  const format = useCameraFormat(device, [{ photoResolution: 'max' }]);
  ...
  format={format}

Relevant log output

com.mrousavy.camera.extensions.CameraDevice_createCaptureSessionKt$createCaptureSession$2$callback$1 in onConfigureFailed at line 45
android.hardware.camera2.impl.CallbackProxies$SessionStateCallbackProxy in lambda$onConfigureFailed$1$android-hardware-camera2-impl-CallbackProxies$SessionStateCallbackProxy at line 64
android.hardware.camera2.impl.CallbackProxies$SessionStateCallbackProxy$$ExternalSyntheticLambda4 in run at line 4
android.os.Handler in handleCallback at line 942
android.os.Handler in dispatchMessage at line 99
android.os.Looper in loopOnce at line 201
android.os.Looper in loop at line 288
android.os.HandlerThread in run at line 67

Camera Device

// I couldn't quite get this because it happened in CI :(

Device

Pixel 7 Pro

VisionCamera Version

3.2.2

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

[ ] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

elliottkember commented 1 year ago

Update: I believe this was a race condition. Adding a 1s timeout between the device being available and the camera showing seems to have resolved this for us.

It seems as though something is trying to create a capture session before the camera is actually ready – with the new improvements in v3 this must have caused something to happen too quickly.

mrousavy commented 1 year ago

Hey, thanks for your research. Two things:

The hdr prop can only be enabled if your format supports HDR. So it should be hdr={format.supportsVideoHDR}. Not always true.
"I believe this was a race condition" - still needs to be fixed! 1s delay is a lot haha. I believe I fixed this in 3.4.0, as I did a ton of refactors around the single lock Mutex there - try upgrading and let me know if that works!

elliottkember commented 1 year ago

@mrousavy Thank you! That hdr advice is good to know. Hopefully that wasn't part of the crash? I would hope that the prop is overridden if the device doesn't support HDR.

And yes, it looks like 3.4.0 / 3.5.0 has fixed the crashing issue with the CaptureSession startup mutex change. Wow the camera is really fast now!

One gotcha that I just found was, if I initialized the camera before format was defined (from useCameraFormat) the camera would fail to initialize, stay blank, and taking photos would error out (camera not ready). I would hope it could update and start the session when format becomes defined. Conditionally rendering the camera on {device && <ReanimatedCamera made it work.

Fortunately in my case I was able to remove the format prop (and HDR) entirely and I believe that will solve it.

mrousavy commented 1 year ago

I would hope that the prop is overridden if the device doesn't support HDR.

well no, I can't just ignore everything and handle all cases - the user is responsible for checkign if HDR is supported or not. otherwise it'd be a blackbox

And yes, it looks like 3.4.0 / 3.5.0 has fixed the crashing issue with the CaptureSession startup mutex change. Wow the camera is really fast now!

Awesome, great to hear! :)

One gotcha that I just found was, if I initialized the camera before format was defined (from useCameraFormat) the camera would fail to initialize, stay blank, and taking photos would error out

Can you create a separate issue for that? That should work...

elliottkember commented 1 year ago

Oops - turns out I just wasn't waiting for onInitialized. My bad! I don't think I had ever waited for that.

well no, I can't just ignore everything and handle all cases - the user is responsible for checkign if HDR is supported or not. otherwise it'd be a blackbox

I was thinking this could just be a separate prop named something like preferHDR? Then we don't have to useCameraFormat to get the format at all. That way I get HDR content if it's available, and not if it's not. That way the component doesn't have to know about the format or use useCameraFormat at all.

In fact, it would be nice if there were optional props (like preferDevice="back" or devices={['back', 'front']}) that could be used in place of hooks. Having to call the hook first means splitting the implementation into two places in the component, sometimes by quite a few lines. In my case, useCameraDevices() is on line 108 and ReanimatedCamera is on line 501 (I know, I know). Keeping those things all together in the JSX would be neat.

Obviously there are great uses for the hook (displaying the format, toggling options etc in the UI) but some apps don't need it.

Not an urgent thing, just a thought!

mrousavy commented 1 year ago

@elliottkember that's exactly why VisionCamera is different than e.g. react-native-camera or other camera libraries.

VisionCamera exposes the full control over device, format and exact controls like HDR to the JS side, and from there on the user has full control over the hardware. APIs like useCameraFormat make this declarative and easy to choose a format, so e.g. this:

const format = useCameraFormat(device, [
  { hdr: true }
])

Is effectively the same as a prefersHDR={true} prop, but it is:

More expressive and precise, it's not "prefers hdr" but instead "find a format that has HDR"
You know if HDR is actually available or not (format.supportsHDR is true) and can show that to the user via a HDR button

And most importantly: It's not a black-box. "Prefers this, prefers that" - what really gets selected? What if you have two formats, one HDR @ 30 FPS and one without HDR @ 60 FPS.

What happens if you pass prefersHDR={true} and prefers60FPS={true}? Which one gets selected?

VisionCamera is powerful because you have full control over everything (hardware sensors, even Frame buffers reading) straight from JS, making it a much better API imo.

elliottkember commented 1 year ago

@mrousavy I'm on board with that! The library is great and very powerful and I am very grateful especially for v3. Thanks for the detailed explanation. I really appreciate the thought that went into it.

In my application I don't need any of the configuration or HDR. But If I really need a super-simple option, I can always make a wrapper (or even a react-native-simple-vision-camera library) that wraps this library with a super simple implementation for use-cases like mine.

IvanByRool commented 11 months ago

Hey,

I'm having, I think, the same issue :

[session/cannot-create-session]: [session/cannot-create-session] Failed to create a Camera Session for Camera 0! This is a Sentry log, it's not a device I have access to.

The user told me he's getting a dark screen on camera

Some context :

This is happening when the user is using the video to process frames (I'm using the MLKit to extract text)
It does work when he's just trying to take a picture
Android 13 / Device : SM-G781B

The code from the frameProcessor

const getPixelFormat = () => {
  return Platform.OS === 'android' ? 'yuv' : 'rgb';
};
const device = useCameraDevice('back');
  const format = useCameraFormat(device, [
    {pixelFormat: getPixelFormat(), videoResolution: {width: 888, height: 420}},
  ]);

<Camera
        style={{
          height: props.height || 100,
        }}
        frameProcessor={frameProcessor}
        onInitialized={() => {
          setIsCameraReady(true);
        }}
        onError={err => {
          Sentry.captureMessage('camera error' + err, 'info'); // this is where the error is captured
        }}
        device={device}
        enableBufferCompression={true}
        video={true}
        fps={format?.maxFps}
        pixelFormat={getPixelFormat()}
        format={format}
        isActive={true}
/>

The code from the Picture camera

const device = useCameraDevice('back');
const format = useCameraFormat(device, [{photoResolution: 'max'}]);

<Camera
        ref={(_ref: Camera) => {
          setCameraRef(_ref);
        }}
        style={previewDimension}
        device={device}
        zoom={device?.neutralZoom}
        photo={true}
        format={format}
        isActive={true}
        enableDepthData={false}
        enableHighQualityPhotos={true}
        enableZoomGesture={false}
        onInitialized={() => setIsCameraReady(true)}
/>

And my package.json

"react": "18.2.0",
"react-native": "0.72.7",
"react-native-reanimated": "^3.5.4",
"react-native-vision-camera": "^3.6.8",
"react-native-worklets-core": "^0.2.4",

Thanks for your help,

Ivan

IvanByRool commented 11 months ago

Hey,

I got new info on this :

This is causing a black screen on Android 13 / Device : SM-G781B


const frameProcessor = useFrameProcessor(frame => {
    'worklet';
    if (props.stopFrameProcessor && !isCameraReady) {
      return;
    }
      console.log('working');
  }, []);

const device = useCameraDevice('back');
  const format = useCameraFormat(device, [
    {pixelFormat: 'yuv', videoResolution: {width: 888, height: 420}},
  ]);

<Camera
        style={{
          height: props.height || 100,
        }}
        frameProcessor={frameProcessor}
        onInitialized={() => {
          console.log('camera is ready');
          setIsCameraReady(true);
        }}
        onError={err => {
          console.log('camera has error');
          Sentry.captureMessage('camera error on visionFrame' + err, 'info');
        }}
        device={device}
        pixelFormat={'yuv'}
        format={format}
        isActive={true}
      />

If i comment frameProcessor={frameProcessor}, then the camera is working.

Maybe I'm wrong, but it seems you have a bug on your lib @mrousavy. Anything else I should try to help you fix this ?

I don't have direct access to the device, all I can do is share a build, and I'm getting feedback within hours.

Let me know if you prefer me to open a new bug,

Thanks; Ivan

IvanByRool commented 11 months ago

Btw it's working with a build on V2.x

pardeep16 commented 10 months ago

set it video as true and it's working.

<Camera style={StyleSheet.absoluteFill} device={device} isActive={true} video={true} onInitialized={()=> console.log('Camera is ready!')} />

mrousavy / react-native-vision-camera