search

mrousavy / react-native-vision-camera

📸 A powerful, high-performance React Native Camera library.
https://react-native-vision-camera.com
MIT License
7.28k stars 1.07k forks source link

🐛 useSkiaFrameProcessor shows black screen with or without frame.render() but useFrameProcessor does not #3034

Open ElianMalessy opened 2 months ago

ElianMalessy commented 2 months ago

What's happening?

When i use the "useSkiaFrameProcessor" even though I have frame.render(), what is rendered is a black screen, instead of the camera feed. When I have the normal "useFrameProcessor" I do not have this issue, and it shows the camera feed. I can draw onto the screen with skia with no problem, and if I do frame.render(paint), that color takes up the whole screen.

Other info: "react-native-reanimated": "~3.10.1" "react-native-worklets-core": "^1.3.3",

This is my babel.config.js:

module.exports = function (api) {
  api.cache(true);
  return {
    presets: [
      ["babel-preset-expo", { jsxImportSource: "nativewind" }],
      "nativewind/babel",
    ],

    plugins: [
      ['react-native-worklets-core/plugin'],
      ['react-native-reanimated/plugin']
    ],
  };
};

Reproduceable Code

const device = useCameraDevice(position);
const pixelFormat = Platform.OS === 'ios' ? 'rgb' : 'yuv';
const format = useMemo(
    () => (device != null ? getBestFormat(device, 720, 1000) : undefined),
    [device],
);

const frameProcessor = useSkiaFrameProcessor(
    frame => {
      'worklet';
      frame.render();
    },
    [plugin, paint],
);

return (
    <View style={styles.container} onTouchEnd={flipCamera}>
      <StatusBar barStyle="light-content" />
      {!hasPermission && <Text style={styles.text}>No Camera Permission.</Text>}
      {hasPermission && device != null && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive={true}
          frameProcessor={frameProcessor}
          format={format}
          pixelFormat={pixelFormat}
        />
      )}
    </View>
);

Relevant log output

06-29 03:22:32.254  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.255  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.270  2169  3141 I CameraView: invokeOnAverageFpsChanged(30.211480362537763)
06-29 03:22:32.317  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.317  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.353  2169  2169 D EGL_emulation: app_time_stats: avg=33.12ms min=10.70ms max=67.84ms count=31
06-29 03:22:32.363  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.363  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.417  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.418  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.419  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.420  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.421  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.431  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.432  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.433  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.434  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.435  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.436  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.437  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.437  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.438  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.439  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.439  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.440  2169  2179 W System  : A resource failed to call HardwareBuffer.close.
06-29 03:22:32.495  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.495  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.513  2169  2396 D EGL_emulation: app_time_stats: avg=30.25ms min=1.78ms max=81.17ms count=30
06-29 03:22:32.757  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.758  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:32.933  2169  2169 E nba.ai  : [SurfaceTexture-0-2169-0] updateAndRelease: EGLConsumer is not attached to an OpenGL ES context
06-29 03:22:32.933  2169  2169 I RNSkia  : updateAndRelease() failed. The exception above can safely be ignored
06-29 03:22:33.271  2169  3141 I CameraView: invokeOnAverageFpsChanged(30.333670374115265)

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "limited",
  "maxZoom": 1,
  "minZoom": 1,
  "maxExposure": 6,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "ultra-wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 999.999985098839,
  "minExposure": -6,
  "name": "10 (BACK) androidx.camera.camera2",
  "hasFlash": false,
  "hasTorch": false,
  "position": "back",
  "id": "10"
}

Device

Pixel 8 emulator API 35

VisionCamera Version

4.3.2

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

maintenance-hans[bot] commented 2 months ago

Guten Tag, Hans here.

[!NOTE] New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time. To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖. Sponsored issues will be prioritized.

ElianMalessy commented 2 months ago

This is the result I get from frame.__skImage.encodeToBase64(), clearly just a completely black image:

iVBORw0KGgoAAAANSUhEUgAABQAAAALQCAIAAABAH0oBAAAAA3NCSVQICAjb4U/gAAAKjklEQVR4nO3BMQEAAADCoPVP7W8GoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIA3NUYAAd1QgUwAAAAASUVORK5CYII=

mrousavy commented 2 months ago

Did you try pixelFormat="rgb"?

ElianMalessy commented 2 months ago

When I do that, I get "Frame Processor Error: Exception in HostFunction: java.lang.Error: Frame has invalid PixelFormat! Only YUV_420_888 is supported. Did you set pixelFormat="yuv"?, js engine: VisionCamera"

mrousavy commented 2 months ago

This error is not thrown in VisionCamera or Skia.

I'm assuming you are using the resize plugin? I don't see you mentioning that in your issue though.

ElianMalessy commented 2 months ago

Sorry about that, I was using the resize plugin. I've removed it an am no longer getting an error, but my screen is still black.

WillyGV commented 1 month ago

Hello, can you try to do this? https://github.com/mrousavy/react-native-vision-camera/issues/2951#issuecomment-2229539912 but without mirror mode. to know if the error is for the same reason.

steq28 commented 2 weeks ago

same here