🐛 Camera is not aligned properly when using `useSkiaFrameProcessor`

[karventhan13]

What's happening?

🐛 [^4.0.0-beta.16] - Camera is not aligned properly when using useSkiaFrameProcessor but it's work with

Reproduceable Code

return (
    <SafeAreaView style={styles.container}>
      {!hasPermission && <Text>No Camera Permission.</Text>}
      {hasPermission && device != null && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive={true}
          photo={true}
          video={true}
          format={format}
          onError={(err)=> { console.log("err **",err)}}
          fps={fps}
          enableFpsGraph={true}
          frameProcessor={frameProcessor}  
          videoHdr={format.supportsVideoHdr}
          photoHdr={format.supportsPhotoHdr}
          enableZoomGesture={false}
        />
      )}
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
});

Relevant log output

none

Camera Device

none

Device

vivo y21

VisionCamera Version

[^4.0.0-beta.16]

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

• [X] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[shaozhenged]

+1

[mrousavy]

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

[mrousavy]

Maybe play around with useSkiaFrameProcessor hook, specifically the render() function in there. If you can figure out what's wrong, please submit a PR. I'm on vacation this week

[Space6188]

This happened when you using resizeMode={'contain'}

Try to delete this from Camera props

[mrousavy]

@Space6188 does it properly display without the resizeMode?

[mrousavy]

I can also reproduce this when using resizeMode="cover"...

[mrousavy]

Try playing around with the code from useSkiaFrameProcessor: https://github.com/mrousavy/react-native-vision-camera/blob/3ff02cc6b4fc34154ec6fb5e650e074e34408e47/package/src/skia/useSkiaFrameProcessor.ts#L165-L175

Maybe we're doing something wrong.

Also, the ImageProxy (which is what the native Frame type holds) contains a [getSensorToBufferTransformMatrix()](https://developer.android.com/reference/androidx/camera/core/ImageInfo#getSensorToBufferTransformMatrix()) method - maybe we can use this Matrix to transform the Frame safely to view dimensions? cc @rodgomesc if you wanna look into this.

[rodgomesc]

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

then you will have zero issues in vc 😅

[rodgomesc]

i tried all resizeMode possibilities, also all fit possibilities, the problem is not there

[mrousavy]

then it's probably the sensor to buffer transform matrix.

[Space6188]

@Space6188правильно ли он отображается без resizeMode?

No,my bad,sorry, it is not working too

[rodgomesc]

i think that i found a fix for this, should send a pr some time tomorrow

[AndyRC6]

In case it helps - I am able to reproduce this by passing in the format prop with a custom videoResolution to the camera. Removing the format prop seems to fix the issue.

Edit: Actually, removing format only fixes the white bar that appears to the left of the preview, and moves it down to the bottom. Which is less noticeable as it looks like that's just where the camera preview ends. But in my case, the preview should take up the entire screen, but doesn't when using the Skia frame processor.

[rodgomesc]

it seems that canvas.drawImage draws the image with a default clip and matrix that are messed by the canvas rotation somehow? that’s confusing since if we use canvas.drawImageRect with the same src/dst props it draws correctly! in my tests it seem to solve the issue

diff --git a/package/src/skia/useSkiaFrameProcessor.ts b/package/src/skia/useSkiaFrameProcessor.ts
index 3feb336..fd69ca1 100644
--- a/package/src/skia/useSkiaFrameProcessor.ts
+++ b/package/src/skia/useSkiaFrameProcessor.ts
@@ -6,8 +6,34 @@ import { VisionCameraProxy, wrapFrameProcessorWithRefCounting } from '../FramePr
 import type { DrawableFrameProcessor } from '../types/CameraProps'
 import type { ISharedValue, IWorkletNativeApi } from 'react-native-worklets-core'
 import { WorkletsProxy } from '../dependencies/WorkletsProxy'
-import type { SkCanvas, SkPaint, SkImage, SkSurface } from '@shopify/react-native-skia'
+import { type SkCanvas, type SkPaint, type SkImage, type SkSurface, ColorType, AlphaType, Skia } from '@shopify/react-native-skia'
 import { SkiaProxy } from '../dependencies/SkiaProxy'
+import { InteractionManager } from 'react-native'
+
+// Mocked Frame Data for testings
+const mockedImageWidth = 1920
+const mockedImageHeight = 1080
+
+const pixels = new Uint8Array(mockedImageWidth * mockedImageHeight * 4)
+pixels.fill(255)
+let i = 0
+for (let x = 0; x < mockedImageWidth; x++) {
+  for (let y = 0; y < mockedImageHeight; y++) {
+    pixels[i++] = (x * y) % 255
+  }
+}
+
+const data = Skia.Data.fromBytes(pixels)
+const image = Skia.Image.MakeImage(
+  {
+    width: mockedImageWidth,
+    height: mockedImageHeight,
+    alphaType: AlphaType.Opaque,
+    colorType: ColorType.RGBA_8888,
+  },
+  data,
+  mockedImageWidth * 4,
+)!

 /**
  * Represents a Camera Frame that can be directly drawn to using Skia.
@@ -155,7 +181,7 @@ export function createSkiaFrameProcessor(

     // Convert Frame to SkImage/Texture
     const nativeBuffer = (frame as FrameInternal).getNativeBuffer()
-    const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
+    // const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)

     return new Proxy(frame as DrawableFrame, {
       get: (_, property: keyof DrawableFrame) => {
@@ -165,11 +191,36 @@ export function createSkiaFrameProcessor(
             'worklet'
             // rotate canvas to properly account for Frame orientation
             canvas.save()
-            const rotation = getRotationDegrees(frame.orientation)
-            canvas.rotate(rotation, frame.width / 2, frame.height / 2)
-            // render the Camera Frame to the Canvas
-            if (paint != null) canvas.drawImage(image, 0, 0, paint)
-            else canvas.drawImage(image, 0, 0)
+
+            const rotationAngle = 0 //getRotationDegrees('portrait')
+            const { width: frameWidth, height: frameHeight } = getPortraitSize(frame)
+
+            // center of the canvas
+            const centerX = frameWidth / 2
+            const centerY = frameHeight / 2
+
+            const currentPaint = paint ?? Skia.Paint()
+
+            // only draw the rect by 40% of the frame size so we can vizualize the whole thing to debug :)
+            const reductionFactor = 0.4
+            const rectWidth = frameWidth * reductionFactor
+            const rectHeight = frameHeight * reductionFactor
+
+            // apply rotation around the center of the canvas
+            canvas.rotate(rotationAngle, centerX, centerY)
+
+            const rectX = centerX - rectWidth / 2
+            const rectY = centerY - rectHeight / 2
+
+            const srcRect = Skia.XYWHRect(0, 0, image.width(), image.height())
+            // Define the destination rectangle on the canvas
+            const dstRect = Skia.XYWHRect(rectX, rectY, rectWidth, rectHeight)
+            // Draw the image on the canvas
+
+            canvas.drawImageRect(image, srcRect, dstRect, currentPaint)

             // restore transforms/rotations again
             canvas.restore()
@@ -185,7 +236,7 @@ export function createSkiaFrameProcessor(
           return () => {
             'worklet'
             // dispose the Frame and the SkImage/Texture
-            image.dispose()
+            // image.dispose() // we do not dispose our mocked image :)
             nativeBuffer.delete()
           }
         }

[!NOTE]
I'm doing some tricks to print the preview at 40% of its original size for better debugging However, at the end of the day, this code just simplifies to:

const srcRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
const dstRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
canvas.drawImageRect(image, srcRect, dstRect, currentPaint);

portrait 0deg	landscape-left 90deg
portrait-upside-down 180deg	landscape-right 270deg

[rodgomesc]

Oh wait, this solves the misalignment of the frame inside the canvas; however, it causes another issue: the image buffer is now rotated 90 degrees clockwise inside a (0 deg portrait) preview for me. I think this is expected since we are lacking orientation support for Android, @mrousavy?

[mrousavy]

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

[rodgomesc]

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android, and in render you are just doing getRotationDegrees(frame.orientation) so i can't visualize the full picture here, of how did you got the image buffers always in portrait without a counter-clockwise rotation of -90deg in the current code that's running in render() {...}

https://github.com/mrousavy/react-native-vision-camera/blob/a53166b01dd08678a354ad81fc04bf9f67bee9f3/package/android/src/main/java/com/mrousavy/camera/frameprocessors/Frame.java#L89-L93

[mrousavy]

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android,

no, frame.orientation is an exception here - this is the sensor relative orientation of the ImageProxy, so this is correct as it is.

[pweglik]

Any news on this? I'm also experiencing this misplacement. I observed that it depends on format and video resolution:

1. Without passing format props to Camera:

   

2. Default format got from useCasmeraFormat():

   const format = useCameraFormat(device, []);
   console.log(format);

   Log:

   
   {"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

<img src="https://github.com/mrousavy/react-native-vision-camera/assets/36445788/e1271238-78cb-412b-80e9-0d1903eb9c9d"  width="280" height="640">

3. 

const format = useCameraFormat(device, [ { videoResolution: { width: 600, height: 400 } }, ]); console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

<img src="https://github.com/mrousavy/react-native-vision-camera/assets/36445788/1e1908dc-9496-4acc-946c-10a743038a89"  width="280" height="640">

4.

const format = useCameraFormat(device, [ { videoResolution: { width: 1200, height: 800 } }, ]); console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

<img src="https://github.com/mrousavy/react-native-vision-camera/assets/36445788/cbe4d179-3c88-44d2-8b3d-8c58b819580c"  width="280" height="640">
![2024-05-08 10 01 42]()

So in case of lower resolutions (default or set manually) only bottom part is cut off, while in higher resolutions entire frame is misplacement. Maybe that somehow helps you

[OleksiiZdaly]

Є новини з цього приводу? Я також відчуваю це неправильне розташування. Я помітив, що це залежить від формату та роздільної здатності відео:

1. Без передачі параметрів формату в камеру:

2. Формат за замовчуванням отримано з useCasmeraFormat():

  const format = useCameraFormat(device, []);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

![2024-05-08 10 01 42]() Таким чином, у разі нижчої роздільної здатності (за замовчуванням або встановленої вручну) обрізається лише нижня частина, тоді як у вищій роздільній здатності зміщується весь кадр. Можливо, це якось тобі допоможе

For some reason, the format parameters don't change anything for me, the behavior is the same as yours in the second example. And the same problem with useSkiaFrameProcessor. On the emulator, the screen is completely black, and on real devices, part of it is. Another strange thing is that the frame itself processes frames rotated 90 degrees, I noticed this when my model started drawing rectangle around object on a black screen turned to the horizontal position. In IOS all ok.

[mrousavy]

Another strange thing is that the frame itself processes frames rotated 90 degrees

That's because of how Cameras work. Sensors are in 90deg orientation. See frame.orientation.

[tomerh2001]

I also encountered this when using useSkiaFrameProcessor:

but useFrameProcessor works normally.

[EnzoDomingues]

I just changed this line

-              canvas.rotate(rotation, frame.width / 2, frame.height / 2)
+              canvas.rotate(rotation, frame.width / 2, frame.height / 1.5)

on useSkiaFrameProcessor.ts Not it's working as expected

[mrousavy]

Created a fix for this here; https://github.com/mrousavy/react-native-vision-camera/pull/2931

Can you guys test if this works for you? :)

[EnzoDomingues]

Created a fix for this here; #2931

Can you guys test if this works for you? :)

@mrousavy Hey, for me, it's even worse I'm using Moto G84

[EnzoDomingues]

@mrousavy I was using this format before

const format = useCameraFormat(device, [
    {
      videoResolution: {
        width: 480,
        height: Platform.OS === 'ios' ? 640 : 720,
      },
    },
  ]);

Using this format, even the colors are changing

[rodgomesc]

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

[EnzoDomingues]

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

[mrousavy]

Hm. Matrixes are really hard to get right.

[rodgomesc]

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

yup, i can confirm, it's worse for me as well

[rodgomesc]

@EnzoDomingues can you confirm if applying this patch https://github.com/mrousavy/react-native-vision-camera/issues/2762#issuecomment-2084282889 the frame keeps centered for you?

[mrousavy]

Try again now - I think it now works :)

[EnzoDomingues]

@mrousavy I added your changes locally and the video is not showing anymore

[EnzoDomingues]

@EnzoDomingues can you confirm if applying this patch #2762 (comment) the frame keeps centered for you?

@rodgomesc, with your patch

[mrousavy]

@EnzoDomingues can you log frame.orientation?

[EnzoDomingues]

@mrousavy sure

 LOG  landscape-right
 LOG  landscape-right
 LOG  landscape-right

[mrousavy]

Ah that's why! Can you try to adjust this code here: https://github.com/mrousavy/react-native-vision-camera/blob/cd183680753837ed5155ca850d7f403ba8ce723c/package/src/skia/useSkiaFrameProcessor.ts#L75-L78

I don't have a phone where it's sensor orientation is landscape-right, so I cannot test this myself. Maybe just switch out the translations, use 0 or whatever - if you find a fix pls let me know.

[mrousavy]

oh lol after posting this I just realized I used the wrong function.... lmao see https://github.com/mrousavy/react-native-vision-camera/commit/38c6983f367524aaefb6bbc2557fe161d7db99ad

now that should work

[EnzoDomingues]

@mrousavy

   // rotate two flips on (0,0) origin and move X + Y into view again
        canvas.translate(0, frame.width)
        canvas.rotate(270, 0, 0)

This works for me instead canvas.translate(frame.height, frame.width) Could you update it?

[mrousavy]

aha interesting... can you send a PR?

[EnzoDomingues]

@mrousavy MR created :)

[tomerh2001]

Is there a release containing this fix?

[mrousavy]

Not yet.

[tomerh2001]

Not yet.

Waiting on this for prod release

[pke]

I wonder @mrousavy why your sample app does not seem to have this issue? My app is in Portrait mode and OS is locked in rotation too but the SKIA preview is a zoomed rotated one.

[mrousavy]

@pke are you using the current main branch? As I just said above; there is no release for this yet. It's on main, but not on npm.

[pke]

Tried to use "react-native-vision-camera": "mrousavy/react-native-vision-camera#bd955b5" in package file, which removes the POD when running bundle exec pod install :/

[opassion]

What's the plan to release this one? @mrousavy