search

mrousavy / react-native-vision-camera

📸 A powerful, high-performance React Native Camera library.
https://react-native-vision-camera.com
MIT License
6.64k stars 983 forks source link

Black screen with useSkiaProcessor 🐛 #2835

Closed DeanGracey closed 2 weeks ago

DeanGracey commented 2 weeks ago

What's happening?

Thanks for a great library! I've been using it nicely until trying to use frame processors and Skia drawing. Everything works well when I use useFrameProcessor, but when I use useSkiaFrameProcessor I just get a black screen. I also use recording in my app and when I record, I can see the video is recorded properly, so it's just the display that's black.

Perhaps I'm misunderstanding how the display is supposed to work, I would like the usual camera to be displayed and I would like to draw lines over it.

Reproduceable Code

const plugin = useTensorflowModel(
    require("../../assets/pose-detection-fast.tflite")
  );

  const paint = Skia.Paint();
  paint.setStrokeWidth(5);
  paint.setColor(Skia.Color("white"));

  const frameProcessor = useSkiaFrameProcessor(
    (frame) => {
      "worklet";
      if (!plugin.model) return;
      const resized = resize(frame, 192, 192);
      const outputs = plugin.model.runSync([resized]);
      const output = outputs[0]; // one output tensor

      const frameWidth = frame.width;
      const frameHeight = frame.height;

      // draw each line on the skeleton
      for (let i = 0; i < lines.length; i += 2) {
        const from = lines[i];
        const to = lines[i + 1];

        const confidence = output[from * 3 + 2];
        if (confidence > 0.5) {
          // frame.drawLine(
          //   (output[from * 3 + 1] as number) * frameWidth,
          //   (output[from * 3] as number) * frameHeight,
          //   (output[to * 3 + 1] as number) * frameWidth,
          //   (output[to * 3] as number) * frameHeight,
          //   paint
          // );
        }
      }
    },
    [plugin, plugin.model, paint]
  );

  let device: CameraDevice | undefined = undefined;
  device = useCameraDevice("front");

<VisionCamera
            style={StyleSheet.absoluteFill}
            device={device}
            isActive={true}
            video={true}
            ref={cameraRef}
            frameProcessor={frameProcessor}
          />

Relevant log output

LOG  Loading react-native-worklets-core...
 LOG  Worklets loaded successfully
 LOG  Installing bindings...
 LOG  Successfully installed!
 LOG  Loading Tensorflow Lite Model 15
 LOG  Resolved Model path: http://localhost:8081/assets/?unstable_path=.%2Fassets/pose-detection-fast.tflite?platform=android&hash=361a28992b649751d6d66cd8cfb1e2a6
 LOG  Model loaded!

These logs continuously log:

2024-05-02 09:20:08.187  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.rawdata.dump"
2024-05-02 09:20:08.187  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.dump"
2024-05-02 09:20:08.187  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.samplerate"
2024-05-02 09:20:08.203 31633-31773 gralloc4                com.swing.swingai.dev                E  Attempt to call unlock*() on an unlocked buffer (0xb4000072074cb240)
2024-05-02 09:20:08.203 31633-31773 Gralloc4                com.swing.swingai.dev                E  unlock(0xb4000072074cb240) failed with 2
2024-05-02 09:20:08.204 31633-31773 Frame                   com.swing.swingai.dev                I  Converting 640 x 480 @ 640 HardwareBuffer...
2024-05-02 09:20:08.207  1459-2884  RequestRegulator        camerahalserver                      I  [onThreadLoop] VSirq : Done(1) SofIdx(8613) res(0) getSofIdx(8611/1) processed(4309<->4308), GUARD(Hal3A:ResC_F3-23.895, ResC_F2-23.895, ResC_F-23.895, ResC3-47.462, ResC2-47.462, Mgr_E-47.462, AFA-47.462, AwbP-47.462, ResC_F4-23.895, Ai3a-47.462, Tone-47.462, Flash-47.462, Lsc-47.462, ResC_F1-23.895, ResC-47.462, Flk-47.462, Ir-47.461, Ae-47.461, Awb-47.462, Stt-47.459, Cac-47.462, MgrS_E-47.459, FlashS-47.459, CacS-47.459, SenS-47.459, Ai3aS-47.459, ToneS-47.459, AwbS-47.459, Sen-47.461, AFAS-47.459, LscS-47.459, FlkS-47.459, AeS-47.459, SttS-47.459, Hal3AFlow:CB_E-47.463, CB_B-47.462, SetAAO-47.459, SetAFO-47.459, Control-47.479, Finish-47.462, Setting-47.465, )
2024-05-02 09:20:08.213  1459-2943  MtkCam/Str...ipe/NextIO camerahalserver                      W  [checkAllDone]Node(fpipe.async) IOReq not used
2024-05-02 09:20:08.218  1459-2879  isp_tuning_custom       camerahalserver                      D  dev_id[2] vendor_id[0x10] map to cameraModuleKey[0]
2024-05-02 09:20:08.218  1459-2878  isp_tuning_custom       camerahalserver                      D  dev_id[2] vendor_id[0x10] map to cameraModuleKey[0]
2024-05-02 09:20:08.219  1459-2851  MtkCam/CamCtrl          camerahalserver                      I  [tryDeQ]  [CHECK::FPS::DeQ]  ( 23.67 /second )
2024-05-02 09:20:08.220  1459-2878  ae_custom_transform     camerahalserver                      D  [calculateLuxIndex], Fno(2.240), MinExpTime_s(0.000084000), MinISO(50), AvgLuma(1365), CurCWR(1341), EVIndex1000x(6109)
2024-05-02 09:20:08.220  1459-2878  ae_custom_transform     camerahalserver                      D  [calculateLuxIndex], MaxEVIndex(11.8662),DeltaEV_10x(57.5725) LuxIndex(172.117)
2024-05-02 09:20:08.220  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.rawdata.dump"
2024-05-02 09:20:08.221  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.dump"
2024-05-02 09:20:08.221  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.samplerate"
2024-05-02 09:20:08.238 31633-31773 gralloc4                com.swing.swingai.dev                E  Attempt to call unlock*() on an unlocked buffer (0xb400006cede577c0)
2024-05-02 09:20:08.239 31633-31773 Gralloc4                com.swing.swingai.dev                E  unlock(0xb400006cede577c0) failed with 2
2024-05-02 09:20:08.240  1459-2884  RequestRegulator        camerahalserver                      I  [onThreadLoop] VSirq : Done(1) SofIdx(8615) res(0) getSofIdx(8613/0) processed(4310<->4309), GUARD(Hal3A:ResC_F3-23.895, ResC_F2-23.895, ResC_F-23.895, ResC3-47.496, ResC2-47.496, Mgr_E-47.496, AFA-47.496, AwbP-47.496, ResC_F4-23.895, Ai3a-47.495, Tone-47.495, Flash-47.495, Lsc-47.495, ResC_F1-23.895, ResC-47.496, Flk-47.496, Ir-47.495, Ae-47.494, Awb-47.495, Stt-47.493, Cac-47.495, MgrS_E-47.493, FlashS-47.493, CacS-47.493, SenS-47.493, Ai3aS-47.493, ToneS-47.493, AwbS-47.493, Sen-47.495, AFAS-47.493, LscS-47.493, FlkS-47.493, AeS-47.493, SttS-47.493, Hal3AFlow:CB_E-47.497, CB_B-47.496, SetAAO-47.492, SetAFO-47.492, Control-47.479, Finish-47.496, Setting-47.499, )
2024-05-02 09:20:08.240 31633-31773 Frame                   com.swing.swingai.dev                I  Converting 640 x 480 @ 640 HardwareBuffer...
2024-05-02 09:20:08.247  1459-2943  MtkCam/Str...ipe/NextIO camerahalserver                      W  [checkAllDone]Node(fpipe.async) IOReq not used
2024-05-02 09:20:08.249 31633-3347  TrafficStats            com.swing.swingai.dev                D  tagSocket(176) with statsTag=0x90000, statsUid=-1
2024-05-02 09:20:08.251  1459-2879  isp_tuning_custom       camerahalserver                      D  dev_id[2] vendor_id[0x10] map to cameraModuleKey[0]
2024-05-02 09:20:08.252  1459-2878  isp_tuning_custom       camerahalserver                      D  dev_id[2] vendor_id[0x10] map to cameraModuleKey[0]
2024-05-02 09:20:08.253  1459-2878  ae_custom_transform     camerahalserver                      D  [calculateLuxIndex], Fno(2.240), MinExpTime_s(0.000084000), MinISO(50), AvgLuma(1365), CurCWR(1344), EVIndex1000x(6109)
2024-05-02 09:20:08.253  1459-2878  ae_custom_transform     camerahalserver                      D  [calculateLuxIndex], MaxEVIndex(11.8662),DeltaEV_10x(57.5725) LuxIndex(172.193)
2024-05-02 09:20:08.253  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.rawdata.dump"
2024-05-02 09:20:08.253  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.dump"
2024-05-02 09:20:08.253  1459-2878  libc                    camerahalserver                      W  Access denied finding property "rgbw.fft.samplerate"

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-right",
  "hardwareLevel": "full",
  "maxZoom": 4,
  "minZoom": 1,
  "maxExposure": 24,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 0,
  "minExposure": -24,
  "name": "1 (FRONT) androidx.camera.camera2",
  "hasFlash": false,
  "hasTorch": false,
  "position": "front",
  "id": "1"
}

Device

Xiaomi 12T

VisionCamera Version

4.0.1

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

DeanGracey commented 2 weeks ago

I've just discovered that with useSkiaFrameProcessor you have to call frame.render() 😅

Thanks!