🐛 useSkiaFrameProcessor makes camera being blank

ppanwin10 commented 3 weeks ago

What's happening?

Hi again, apologies for bringing up multiple issues consecutively, but I'm really in need of your assistance.

Everything seems to be functioning properly except when I apply useSkiaFrameProcessor to the camera. The camera screen goes blank.

I tried the exact same process in the example app, and it worked as expected. I've compared the setup in my app with the example app, and everything appears to be identical. It's possible that I've made a mistake somewhere, but I'm reaching out to see if you have any insights into what I might have done wrong.

Reproduceable Code

pacakge.json
"react-native-vision-camera": "^4.0.3",
"react-native-worklets-core": "^1.2.0",
"@shopify/react-native-skia": "^1.2.3",

const frameProcessor = useSkiaFrameProcessor(frame => {
    'worklet';
    frame.render();

    const centerX = frame.width / 2;
    const centerY = frame.height / 2;
    const rect = Skia.XYWHRect(centerX, centerY, 150, 150);
    const paint = Skia.Paint();
    paint.setColor(Skia.Color('red'));
    frame.drawRect(rect, paint);
}, []);

const format = useCameraFormat(device, [
    {fps: DEFAULT_FPS},
    {photoAspectRatio: camera_ratio},
    {photoResolution: 'max'},
    {photoHdr: true},
    {videoAspectRatio: camera_ratio},
    {videoResolution: 'max'},
    {videoHdr: true},
]);

<ReanimatedCameraView
          ref={camera}
          device={device}
          isActive={camera_isActive}
          style={styles.container}
          photo={true}
          video={true}
          audio={false}
          orientation={cameraOrientation}
          animatedProps={animatedProps}
          format={format}
          resizeMode={'cover'}
          frameProcessor={frameProcessor}
          photoHdr={format?.supportsPhotoHdr}
          videoHdr={format?.supportsVideoHdr}
          photoQualityBalance={'quality'}
          fps={fps}
          onError={onError}
/>

Relevant log output

17:16:20.390: [info] 📸 VisionCamera.didSetProps(_:): Updating 30 props: [onInitialized, alignItems, cameraId, enableBufferCompression, preview, onStarted, onCodeScanned, collapsable, justifyContent, photoHdr, position, isActive, video, resizeMode, fps, onStopped, onError, onViewReady, enableFrameProcessor, videoHdr, format, orientation, audio, height, width, zoom, photoQualityBalance, exposure, photo, onShutter]
17:16:20.392: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:16:20.401: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
17:16:20.401: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Input Device...
17:16:20.401: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:6...
17:16:20.409: [info] 📸 VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
17:16:20.409: [info] 📸 VisionCamera.configureOutputs(configuration:): Configuring Outputs...
17:16:20.409: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Photo output...
17:16:20.416: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Video Data output...
17:16:20.418: [info] 📸 VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
17:16:20.419: [info] 📸 VisionCamera.configureFormat(configuration:device:): Configuring Format (2112x1188 | 1920x1080@60.0 (ISO: 32.0..3072.0))...
17:16:20.422: [info] 📸 VisionCamera.configureFormat(configuration:device:): Successfully configured Format!
17:16:20.427: [info] 📸 VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={false})
17:16:20.427: [info] 📸 VisionCamera.getPixelFormat(for:): Using PixelFormat: 420f...
'user login status: ', true
17:16:20.446: [info] 📸 VisionCamera.onCameraStarted(): Camera started!
17:16:20.446: [info] 📸 VisionCamera.onSessionInitialized(): Camera initialized!
17:16:20.446: [info] 📸 VisionCamera.configure(_:): Beginning AudioSession configuration...
17:16:20.446: [info] 📸 VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
17:16:20.446: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
17:16:20.450: [info] 📸 VisionCamera.configure(_:): Committed AudioSession configuration!
17:16:20.451: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
17:16:20.451: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:16:20.453: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
17:16:20.615: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!
{ isRecordingWithFlash: false, averageFpsSamples: [] }
17:16:20.663: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:16:20.663: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:16:20.664: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)

17:19:43.196: [info] 📸 VisionCamera.didSetProps(_:): Updating 30 props: [onInitialized, alignItems, cameraId, enableBufferCompression, preview, onStarted, onCodeScanned, collapsable, justifyContent, photoHdr, position, isActive, video, resizeMode, fps, onStopped, onError, onViewReady, enableFrameProcessor, videoHdr, format, orientation, audio, height, width, zoom, photoQualityBalance, exposure, photo, onShutter]
17:19:43.196: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:19:43.197: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
17:19:43.197: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Input Device...
17:19:43.197: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:6...
17:19:43.206: [info] 📸 VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
17:19:43.206: [info] 📸 VisionCamera.configureOutputs(configuration:): Configuring Outputs...
17:19:43.206: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Photo output...
17:19:43.209: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Video Data output...
17:19:43.210: [info] 📸 VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
17:19:43.210: [info] 📸 VisionCamera.configureFormat(configuration:device:): Configuring Format (2112x1188 | 1920x1080@60.0 (ISO: 32.0..3072.0))...
17:19:43.212: [info] 📸 VisionCamera.configureFormat(configuration:device:): Successfully configured Format!
17:19:43.212: [info] 📸 VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={false})
17:19:43.212: [info] 📸 VisionCamera.getPixelFormat(for:): Using PixelFormat: 420f...
17:19:43.215: [info] 📸 VisionCamera.onSessionInitialized(): Camera initialized!
17:19:43.215: [info] 📸 VisionCamera.configure(_:): Beginning AudioSession configuration...
17:19:43.215: [info] 📸 VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
17:19:43.215: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
17:19:43.216: [info] 📸 VisionCamera.configure(_:): Committed AudioSession configuration!
17:19:43.218: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!

17:21:44.867: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:44.867: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:44.868: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
17:21:44.904: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:44.904: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:44.904: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
17:21:44.997: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:44.997: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:44.997: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
17:21:45.382: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:45.382: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:45.383: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)

(ADVICE) View #247 of type RCTView has a shadow set but cannot calculate shadow efficiently. Consider setting a solid background color to fix this, or apply the shadow to a more specific component.
(ADVICE) View #247 of type RCTView has a shadow set but cannot calculate shadow efficiently. Consider setting a solid background color to fix this, or apply the shadow to a more specific component.
[I] <MMKV_OSX.cpp:99::setIsInBackground> g_isInBackground:0
[I] <libMMKV.mm:313::+[MMKV didBecomeActive]> isInBackground:0
17:21:49.806: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
17:21:49.806: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:49.808: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
17:21:50.253: [info] 📸 VisionCamera.onCameraStarted(): Camera started!
17:21:50.289: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:50.289: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:50.290: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
<<<< FigCaptureSessionRemote >>>> Fig assert: "err == 0 " at bail (FigCaptureSessionRemote.m:866) - (err=-12784)
[I] <MMKV_OSX.cpp:99::setIsInBackground> g_isInBackground:1
[I] <libMMKV.mm:308::+[MMKV didEnterBackground]> isInBackground:1
17:21:51.390: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [isActive]
17:21:51.390: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:51.390: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
[I] <libMMKV.mm:301::-[MMKV onMemoryWarning]> cleaning on memory warning mmkv.default
[I] <MMKV.cpp:307::clearMemoryCache> clearMemoryCache [mmkv.default]
[I] <MemoryFile.cpp:104::close> closing fd[0x15], /var/mobile/Containers/Data/Application/A90E070A-F05A-469B-9785-B47EDD2E7D22/Documents/mmkv/mmkv.default
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0507 17:21:51.708926 1843605504 JSIExecutor.cpp:348] Memory warning (pressure level: TRIM_MEMORY_RUNNING_CRITICAL) received by JS VM, running a GC

17:21:53.775: [info] 📸 VisionCamera.didSetProps(_:): Updating 1 props: [orientation]
17:21:53.775: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
17:21:53.776: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: true, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)

Camera Device

{
  "formats": [],
  "isMultiCam": true,
  "maxZoom": 123.75,
  "hasTorch": true,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:6",
  "minFocusDistance": 12,
  "name": "Back Dual Wide Camera",
  "minExposure": -8,
  "hasFlash": true,
  "minZoom": 1,
  "supportsRawCapture": false,
  "neutralZoom": 2,
  "physicalDevices": [
    "ultra-wide-angle-camera",
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "sensorOrientation": "landscape-right",
  "maxExposure": 8,
  "supportsLowLightBoost": false,
  "hardwareLevel": "full",
  "position": "back"
}

Device

iPhont 11 IOS17.4.1

VisionCamera Version

4.0.3

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

mrousavy commented 3 weeks ago

Hm, did you install Reanimated?

ppanwin10 commented 3 weeks ago

Hi Marc, yes, I do have it installed. "react-native-reanimated": "^3.6.0",

Hm, did you install Reanimated?

mrousavy commented 3 weeks ago

Hm, I don't know. frame.render() should be responsible for showing the Frame.

Does e.g. a red rectangle appear if you render it on screen?

ppanwin10 commented 3 weeks ago

it does not appear in my app but it is working in example app. Anyway, I will keep debug this, you can close this issue. Thank you

islemsyw commented 3 weeks ago

Hello, Can you please help me resolve this issue, when i'm using the frame porcessor and write frame.render(), I get the problem "Property 'render' does not exist on type 'Frame'.

wuguishifu commented 3 weeks ago

@islemsyw are you using the useSkiaFrameProcessor hook? I was getting that a while ago until I realized I was still using the useFrameProcessor hook instead of the Skia one.

islemsyw commented 2 weeks ago

I tried with useSkiaFrameProcessor hook but got this problem "react-native-vision-camera"' has no exported member named 'useSkiaFrameProcessor'. Did you mean 'useFrameProcessor'?" Is it because I'm trying on android?

ppanwin10 commented 2 weeks ago

I tried with useSkiaFrameProcessor hook but got this problem "react-native-vision-camera"' has no exported member named 'useSkiaFrameProcessor'. Did you mean 'useFrameProcessor'?" Is it because I'm trying on android?

did you upgrade vision camera package?

mrousavy / react-native-vision-camera