DarkShtir
commented
5 months ago Same issue. But my version is "4.2.0"
Open khlling opened 5 months ago
DarkShtir
commented
5 months ago Same issue. But my version is "4.2.0"
khlling
commented
5 months ago I've updated my version to "4.2.0" also. By chance, I tipped my phone from portrait to landscape whilst using the front camera and it started working. As soon as I tip it back to portrait it goes black. Try that out @DarkShtir.
I assume therefore it's something to do with the new orientation changes in the recent release. Will have an investigate of the latest changes
DarkShtir
commented
4 months ago @khlling this works, but the same time is not easy and comfortable to use. I Downgraded to "4.0.5" and use custom patch to transform, rotate and align canvas in skia fame processor I'll try get time for additional investigation of this issue
khlling
commented
4 months ago @DarkShtir Agreed I'm sure it's not the intended behaviour.
What changes did you have to make in 4.0.5 to get it to work? Can you share your patch files?
DarkShtir
commented
4 months ago diff --git a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
index 30efc01..9468756 100644
--- a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
+++ b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
@@ -9,6 +9,7 @@ import type { SkCanvas, SkPaint, SkImage, SkSurface } from '@shopify/react-nativ
import { SkiaProxy } from '../dependencies/SkiaProxy'
import { withFrameRefCounting } from '../frame-processors/withFrameRefCounting'
import { VisionCameraProxy } from '../frame-processors/VisionCameraProxy'
+import { Dimensions, Platform } from 'react-native'
/**
* Represents a Camera Frame that can be directly drawn to using Skia.
@@ -166,11 +167,48 @@ export function createSkiaFrameProcessor(
'worklet'
// rotate canvas to properly account for Frame orientation
canvas.save()
- const rotation = getRotationDegrees(frame.orientation)
- canvas.rotate(rotation, frame.width / 2, frame.height / 2)
- // render the Camera Frame to the Canvas
- if (paint != null) canvas.drawImage(image, 0, 0, paint)
- else canvas.drawImage(image, 0, 0)
+ // BUG in Andoid ⬇️
+ if(Platform.OS==='ios'){
+ const rotation = getRotationDegrees(frame.orientation)
+
+ canvas.rotate(rotation, frame.width / 2, frame.height / 2)
+ // render the Camera Frame to the Canvas
+ if (paint != null) canvas.drawImage(image, 0, 0, paint)
+ else canvas.drawImage(image, 0, 0)
+ }
+ // BUG ⬆️
+
+ // FIX on Android⬇️
+ if(Platform.OS==='android'){
+ const rotationAngle = getRotationDegrees(frame.orientation)
+
+ const { width: frameWidth, height: frameHeight } = getPortraitSize(frame)
+
+ // center of the canvas
+ const centerX = image.width() / 2
+ const centerY = image.height() / 2
+
+ const currentPaint = paint ?? Skia.Paint()
+
+ const alphaX = Math.abs((image.width()-frameWidth)/2);
+ const alphaY = Math.abs((image.height()-frameHeight)/2);
+
+
+ const srcRect = Skia.XYWHRect(0, 0, image.width(),image.height())
+ let dstRect = Skia.XYWHRect(0,0, image.width(),image.height())
+ // Define the destination rectangle on the canvas
+ if (frame.orientation==='landscape-right') {
+ dstRect = Skia.XYWHRect(-alphaX,-alphaY, image.width(),image.height())
+ } else if (frame.orientation==='landscape-left') {
+ dstRect = Skia.XYWHRect(alphaX,alphaY, image.width(),image.height())
+ }
+ // Draw the image on the canvas
+
+ canvas.rotate(rotationAngle, centerX, centerY)
+
+ canvas.drawImageRect(image, srcRect, dstRect, currentPaint)
+ }
+ // FIX ⬆️
// restore transforms/rotations again
canvas.restore()
No problem, is local fix for project, to use skiaFrameProcessor But it isn't great, and has bugs, for example reverse rotations when using front camera. But now this fix is resolving our issues. Hope this help you.
tsnguyenducphuong
commented
4 months ago I had the same blank screen issue on Android 14 (Samsung A53) and resolved the issue by changing minSdkVersion from 24 to 26 in android/build.gradle.
tsnguyenducphuong
commented
4 months ago Also add the following would help: const pixelFormat = Platform.OS === 'ios' ? 'rgb' : 'yuv';
mrousavy
commented
4 months ago @tsnguyenducphuong wait I thought VisionCamera/Skia will throw an error if minSdkVersion is not 26 or higher? Need to double-check with @wcandillon, but we should definitely throw very concise errors in such cases instead of returning null and just logging the errors to the console..
mrousavy
commented
4 months ago Either way, the issue from OP (@khlling) is not related to minSdkversion as it renders fine on back camera, but not on front. I think this has to do with the orientation/matrix code - @khlling can you play around with this function to figure out where it goes wrong? https://github.com/mrousavy/react-native-vision-camera/blob/90be06b5534ecddb9b8157881aaea0ed49173cb5/package/src/skia/useSkiaFrameProcessor.ts#L61-L82
I unfortunately don't have the free time to look into this right now, but if you find a solution in there I'd appreciate a PR! Thanks
mrousavy
commented
4 months ago Maybe it'd also be helpful if you posted device.sensorOrientation as well as frame.orientation of both the back and the front camera here just so we know what values we're dealing with.
khlling
commented
4 months ago @tsnguyenducphuong wait I thought VisionCamera/Skia will throw an error if
minSdkVersionis not 26 or higher? Need to double-check with @wcandillon, but we should definitely throw very concise errors in such cases instead of returning null and just logging the errors to the console..
I can confirm that VisionCamera/Skia does throw an error for this. I had to update the minSdkVersionon the back of the build error.
Post this update I'm getting a blank front camera preview.
khlling
commented
4 months ago Either way, the issue from OP (@khlling) is not related to minSdkversion as it renders fine on back camera, but not on front. I think this has to do with the orientation/matrix code - @khlling can you play around with this function to figure out where it goes wrong? https://github.com/mrousavy/react-native-vision-camera/blob/90be06b5534ecddb9b8157881aaea0ed49173cb5/package/src/skia/useSkiaFrameProcessor.ts#L61-L82
I unfortunately don't have the free time to look into this right now, but if you find a solution in there I'd appreciate a PR! Thanks
No worries @mrousavy let me do a bit of digging. Thanks for giving me a hint of where to start looking!
Will update this issue with logs and investigation.
tsnguyenducphuong
commented
4 months ago In my case, I had first the blank screen issue then the black screen and solved with the following steps: (I am using Vision Camera version 4.2.1)
Solve black screen:
wcandillon
commented
4 months ago @mrousavy it's interesting because in the video use-case the error is clean and concise, I assume it would be the same for the vision-camera integration? Could it be because of an older version of Skia (I did improve this recently I think)
mrousavy
commented
4 months ago True, maybe an old RN Skia version! I'll do more research thanks
mfkrause
commented
4 months ago I'm experiencing the same bug, but on iOS. When a Skia frame processor is connected, the camera stays black as long as the iPhone is in portrait. Some observations:
When I rotate the iPhone to landscape (which our UI doesn't support), the camera appears, but is rotated by 90 degrees. On iPad, where we support both portrait and landscape mode for the UI, everything works as expected, though.
When I set outputOrientation to preview, the camera frame stays visible, but stays locked in its 90 degrees rotated position (no matter whether or not the device's screen orientation lock is enabled). Setting it to portrait makes the camera preview stay black permanently (in every device orientation).
I'm not sure if that's expected behavior for apps that don't support landscape mode, but I've also noticed onPreviewOrientationChanged never sends any events and stays locked on portrait.
onOutputOrientation fires events with an orientation that's rotated 90 degrees, too. For example, if the device is rotated to portrait mode, this fires an event with landscape-left. Rotating it to portrait upside down gives landscape-right. This issue might possibly be related to #2984.
From within the frame processor, logging frame.orientation leads to landscape-left when the device is actually in portrait; portrait when the device is actually in landscape left or portrait upside down; and portrait-upside-down when the device is in landscape right. When outputOrientation is set to preview, this will always log portrait-upside-down.
All in all, there seems to be something going wrong with detecting the device's orientation, leading to wrong transforms on the Skia frame.
I just rebuilt the app with the newest react-native-skia release (1.3.4) which didn't fix it, unfortunately.
Edit: My iPhone runs on iOS 17 while my iPad still runs on iOS 16. So, if this is related to #2972, this might be why it works fine on the iPad.
mfkrause
commented
4 months ago After I digged a little into the different orientation values provided by the library, I realized most of my previous comment is actually the expected behavior and the problem must indeed lie in the frame rotation in useSkiaFrameProcessor.ts. I played around a little and came up with this:
switch (frame.orientation) {
case 'portrait':
case 'portrait-upside-down':
canvas.scale(frame.height / frame.width, frame.height / frame.width)
canvas.rotate(90,0,0)
canvas.translate(0, -(frame.height * 2))
break
case 'landscape-left':
case 'landscape-right':
canvas.translate(frame.height, 0)
canvas.rotate(90, 0, 0)
break
default:
throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
}This seems to work great, however, two caveats:
scale() values aren't quite right yet, as the frame still seems to extend beyond the device borders when the device is in landscape.So, this isn't even close to a full fix or PR ready yet, just some observations.
WillyGV
commented
3 months ago Hello, I have the same problem, and you can solve it by changing the logic of the landscape-left case:
diff --git a/package/src/skia/useSkiaFrameProcessor.ts b/package/src/skia/useSkiaFrameProcessor.ts
index 10888109..16b667a2 100644
--- a/package/src/skia/useSkiaFrameProcessor.ts
+++ b/package/src/skia/useSkiaFrameProcessor.ts
@@ -95,8 +95,8 @@ function withRotatedFrame(frame: Frame, canvas: SkCanvas, previewOrientation: Or
break
case 'landscape-left':
// rotate two flips on (0,0) origin and move X + Y into view again
- canvas.translate(frame.height, frame.width)
- canvas.translate(270, 0)
+ canvas.translate(0, frame.width)
+ canvas.rotate(270, 0, 0)
break
case 'portrait-upside-down':
// rotate three flips on (0,0) origin and move Y into view againbut I run into the problem that the camera creates a mirror effect, and I can't solve it. I already tried to add the attribute to the
<camera
isMirrored={true}
/>but the image cannot be reversed
mrousavy
commented
3 months ago but I run into the problem that the camera creates a mirror effect, and I can't solve it. I already tried to add the attribute to the
<camera isMirrored={true} />but the image cannot be reversed
You want to disable mirroring, but pass isMirrored={true}? If you want to disable it you need to set isMirroring={false}...
WillyGV
commented
3 months ago I'm sorry @mrousavy. What I wanted to say is that the mirror effect happens to me in the preview, I saw that in the code and in the documentation that isMirrored does not work for the preview, also I am only working on Android for now and in the Google documentation, the Preview class, Does not support SetMirrorMode for android versions prior to API 32, I think the problem is in Skia's canvas.
I tried isMirrored={false} and isMirrored={true} in the camera component, I haven't found out if it can be changed in the preview
mrousavy
commented
3 months ago Ah, so the Preview is not mirrored with a Skia Frame Processor, but is properly mirrored without a Skia FP - is that what you're saying?
Azmye
commented
3 months ago is there a workaround for this bug ?
JulianKingman
commented
2 months ago Seeing this in iOS, running in an expo development build
"react-native-vision-camera": "^4.3.2",
"@shopify/react-native-skia": "1.2.3",
"react-native-reanimated": "~3.10.1",I'm in early days of development, so it's not a show-stopper at the moment, let me know if I can provide any useful info.
mrousavy
commented
2 months ago you can play around with this function to figure out where it goes wrong: https://github.com/mrousavy/react-native-vision-camera/blob/90be06b5534ecddb9b8157881aaea0ed49173cb5/package/src/skia/useSkiaFrameProcessor.ts#L61-L82
raslankiwan
commented
2 months ago I was facing this issue on iPad, the screen was cutting off in both landscape left and right, and in upside down the whole camera was giving black screen, I made this adjustment and it worked perfectly in all cases (no need to change rotation configuration)
switch (orientation) {
case 'portrait':
canvas.translate( 0,frame.height / 2)
break
case 'landscape-left':
// rotate two flips on (0,0) origin and move X + Y into view again
canvas.translate(0, frame.width)
canvas.rotate(270, 0, 0)
break
case 'portrait-upside-down':
// rotate three flips on (0,0) origin and move Y into view again
canvas.translate(frame.width, frame.height * 1.5)
canvas.rotate(180, 0, 0)
break
case 'landscape-right':
// rotate one flip on (0,0) origin and move X into view again
canvas.translate(frame.height, 0)
canvas.rotate(90, 0, 0)
break
default:
throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
}
sjaiswalgit
commented
2 months ago Face the same issue, found the cause that front camera actaully do a lateral inversion because its need to show picture just like a mirror . You can verify this by drawing something on canvas and changing camera from back to front
Viktor-as
commented
2 months ago I was facing this issue on iPad, the screen was cutting off in both landscape left and right, and in upside down the whole camera was giving black screen, I made this adjustment and it worked perfectly in all cases (no need to change rotation configuration)
switch (orientation) { case 'portrait': canvas.translate( 0,frame.height / 2) break case 'landscape-left': // rotate two flips on (0,0) origin and move X + Y into view again canvas.translate(0, frame.width) canvas.rotate(270, 0, 0) break case 'portrait-upside-down': // rotate three flips on (0,0) origin and move Y into view again canvas.translate(frame.width, frame.height * 1.5) canvas.rotate(180, 0, 0) break case 'landscape-right': // rotate one flip on (0,0) origin and move X into view again canvas.translate(frame.height, 0) canvas.rotate(90, 0, 0) break default: throw new Error(`Invalid frame.orientation: ${frame.orientation}!`) }
thanks @raslankiwan , I had also this issue on Android phone, in portrait view I was getting black screen. In landscape it was ok.
I added your changes in node_modules\react-native-vision-camera\src\skia\useSkiaFrameProcessor.ts and now in portrait I also get camera view
softgenicsShubham
commented
1 month ago @raslankiwan same issues in the --version 4.5.3 also, but will fixed with your changes.
switch (orientation) {
case 'portrait':
canvas.translate( 0,frame.height / 2)
break
case 'landscape-left':
// rotate two flips on (0,0) origin and move X + Y into view again
canvas.translate(0, frame.width)
canvas.rotate(270, 0, 0)
break
case 'portrait-upside-down':
// rotate three flips on (0,0) origin and move Y into view again
canvas.translate(frame.width, frame.height * 1.5)
canvas.rotate(180, 0, 0)
break
case 'landscape-right':
// rotate one flip on (0,0) origin and move X into view again
canvas.translate(frame.height, 0)
canvas.rotate(90, 0, 0)
break
default:
throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
}const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet';
frame.render();
const centerX = frame.width / 2;
const centerY = frame.height / 2;
const rect = Skia.XYWHRect(centerX, centerY, 150, 150);
const paint = Skia.Paint();
paint.setColor(Skia.Color('red'));
frame.drawRect(rect, paint);
}, []);With the above changes, I tried drawing using Skia. However, I am seeing some unexpected behavior on the screen. When the camera is set to the front position, the drawing appears in a different place, and when the camera is set to the back position, the drawing appears in yet another different position. I even tried using isMirror with both true and false values.
boiboif
commented
3 weeks ago Any update?
rachellauyui
commented
1 week ago Hey, any update? Same problem here, black screen when using front camera, but all fine with back camera, both in android and ios.
But what is crazy is that once I transform it { scaleX: 1} in style, meaning did not transform anything, it works in iOS, but not android.... so it seems something in the native code of front camera that cause the issue, probably because front camera needs to mirror (flip left to right) in order to make it looks and works like mirror so that people can really use it to do make-up /shave, etc.
My code in case anybody interested (only work in iOS): <Camera style={{ ...StyleSheet.absoluteFillObject, transform: [{ scaleX: 1 }], }} device={device} isActive={isActive} frameProcessor={frameProcessor} camera={true} ref={camera} />
ivgawev
commented
1 week ago switch (orientation) { case 'portrait': canvas.translate( 0,frame.height / 2) break case 'landscape-left': // rotate two flips on (0,0) origin and move X + Y into view again canvas.translate(0, frame.width) canvas.rotate(270, 0, 0) break case 'portrait-upside-down': // rotate three flips on (0,0) origin and move Y into view again canvas.translate(frame.width, frame.height * 1.5) canvas.rotate(180, 0, 0) break case 'landscape-right': // rotate one flip on (0,0) origin and move X into view again canvas.translate(frame.height, 0) canvas.rotate(90, 0, 0) break default: throw new Error(`Invalid frame.orientation: ${frame.orientation}!`) }
Hi.
Are you guys planning update the package with this fix? Because it worked very well on my device...
frodriguez-hu
commented
3 days ago switch (orientation) { case 'portrait': canvas.translate( 0,frame.height / 2) break case 'landscape-left': // rotate two flips on (0,0) origin and move X + Y into view again canvas.translate(0, frame.width) canvas.rotate(270, 0, 0) break case 'portrait-upside-down': // rotate three flips on (0,0) origin and move Y into view again canvas.translate(frame.width, frame.height * 1.5) canvas.rotate(180, 0, 0) break case 'landscape-right': // rotate one flip on (0,0) origin and move X into view again canvas.translate(frame.height, 0) canvas.rotate(90, 0, 0) break default: throw new Error(`Invalid frame.orientation: ${frame.orientation}!`) }Hi.
Are you guys planning update the package with this fix? Because it worked very well on my device...
It worked well on mine too, I made a patch on my app. But I am no t sure if it is the proper way of handling it
What's happening?
When calling frame.render() within useSkiaFrameProcessor on an Android phone, the captured frames render as expected when the camera position is set to 'back'. However, setting the camera position to 'front' results in a black screen.
Omitting useSkiaFrameProcessor (i.e., not passing a frame processor as a prop to ReanimatedCamera) renders as expected Recording and playing back the video output from the front camera works fine, indicating the frames are captured correctly but not rendered properly in real-time.
Reproduceable Code
Relevant log output
Camera Device
Device
Pixel 6 (Android 14)
VisionCamera Version
4.1.0
Can you reproduce this issue in the VisionCamera Example app?
Yes, I can reproduce the same issue in the Example app here
Additional information