🐛 After taking a photo with the front camera, the resulting photo is rotated 90 degrees on Samsung

[dereklance]

What's happening?

When I take a photo with the front camera, the resulting PhotoFile is rotated 90 degrees counter-clockwise. The output logs from running the example app show that the orientation is 'landscape-left', when it should use the device orientation which is portrait (or at least use the orientation='portrait' prop). This only happens on Android.

Reproduceable Code

const device = useCameraDevice('front');
const isForeground = useIsForeground();

return (
    <Camera
      style={StyleSheet.absoluteFill}
      photo
      device={device}
      ref={ref}
      enableHighQualityPhotos
      zoom={device.neutralZoom}
      isActive={isForeground && isActive}
      orientation='portrait'
    >
      {children}
    </Camera>
  );

Relevant log output

Media captured! {"isMirrored":true,"path":"/data/user/0/com.mrousavy.camera.example/cache/mrousavy3169337114776682149.jpg","isRawPhoto":false,"height":1468,"orientation":"landscape-left","width":3264}

Camera Device

{
  "sensorOrientation": "landscape-right",
  "hardwareLevel": "full",
  "maxZoom": 8,
  "minZoom": 1,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": true,
  "isMultiCam": false,
  "name": "BACK (0)",
  "hasFlash": true,
  "hasTorch": true,
  "position": "back",
  "id": "0"
}

Device

Samsung Galaxy A23 (SM-A235F)

VisionCamera Version

3.3.0

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• [X] I am using Expo
• [ ] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[mrousavy]

Hi.

Please share the photo.

Orientation is not implemented yet, see https://github.com/mrousavy/react-native-vision-camera/issues/1891

[zeohera]

https://github.com/mrousavy/react-native-vision-camera/assets/47137794/cc15829d-9b50-4426-a4f4-df50200c58f3

I have the same issue but only on front camera

[mrousavy]

Yea I think this would be fixed with #1891.

[mrousavy]

Can you try changing this value here to see what works? https://github.com/mrousavy/react-native-vision-camera/blob/e8ae11e30b4fb4e25a38fa589b27229c18c03171/package/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt#L230

Try just

val orientation = Orientation.PORTRAIT

or Orientation.LANDSCAPE_LEFT or whatever - just let me know which of those values work (or if they change anything at all)

Maybe the code in here needs special handling for Samsung: https://github.com/mrousavy/react-native-vision-camera/blob/e8ae11e30b4fb4e25a38fa589b27229c18c03171/package/android/src/main/java/com/mrousavy/camera/parsers/Orientation.kt#L19-L33

Either way, I would fix that with #1891 - for now, I don't have the time

[stefan-schweiger]

@mrousavy I applied val orientation = Orientation.PORTRAIT via patch-package and on my OnePlus 6T the image is now upside down. So it seems like there is no clear cut handling for this on Android.

[mrousavy]

@stefan-schweiger I see - what's the sensor orientation of that Camera Device on your phone then?

val sensorOrientation = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!! 

Can you tell me the value of this?

Also, your View is in Portrait only right? No landscape

[isinuyk]

@mrousavy have the same problem on the Samsung A11 when taking picture using front camera.

in my case this line val sensorOrientation = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!! returns 270

I've tried to change the orientation as you mentioned above to Orientation.LANDSCAPE_LEFT/PORTRAIT/... but nothing changed

P.S. The interesting thing is: that I was also testing it with my Android Emulator (Pixel 2, API 34) and there was the same rotation bug. but, I was able to resolve it there simply by adding orientation='portrait' to the <Camera props. So, definitely there is something wrong with Samsung

RNVC 3.6.3

[mrousavy]

Yea, Samsung for some reason always suck at building consistent Camera APIs.

I had to add a special workaround for 60 FPS on Samsung, and apparently Orientation is the same thing. See this blog post.

Again, this can be solved with the Orientation feature (✨ Implement Orientation ($8,000) #1891), so if you want to support this consider sponsoring there so we can reach that goal :)

[vishaljak]

Noticed the same issues with Pixel 7s (haven't tested but might be the same for other pixel generations), only happens when image is captured using the front camera

[dipankur007]

@mrousavy Noticed the same issue on Realme X3 device as well. Only happening while using front camera.

[isinuyk]

I've been investigating this bug for a while and found the root of the problem. So, the first thing that you need to know is that isMirroredvalue is strictly defined as true if the current device is a Front camera.

Then we are using this isMirroredvariable inside of the writePhotoToFilefunction after the picture was taken.

And there is the main thing. When we have isMirrored = true in that case we will create the Matrix() of our image and scale it to flip it back into the correct view. In the case of the back camera we skip this phase and just write the byte array of this file.

For some reason when we are creating this Matrix() it is rotating our image to the camera sensor orientation. For example, in my case, the front camera orientation is 270 and my resulting image was always rotated by 270 degrees. So, all I did was just not only flip the matrix horizontally as it was already there but additionally rotate the matrix for the value of the sensor orientation.

So, this is how my patch looks like at the end.

react-native-vision-camera+3.6.4.patch

I've tested it on Samsung A11, Motorola g72 - Android 12, and Pixel 3

@mrousavy what do you think? maybe it is worth adding this as a patch to next versions or should we wait for the global Orientation feature?

[mrousavy]

@isinuyk yea this is the thing - this is the function that needs careful adjustment to correctly respect Orientation in all cases - on Huawei, Samsung, Google phones, as well as for front and back camera, mirrored and not mirrored.

Many cases to consider and to find a solution that works everywhere we probably need a bit more handling in there.

So this is part of the Orientation feature imo.

Does your patch still mirror selfies correctly on Pixel phones?

[isinuyk]

@mrousavy Unfortunately, I don't have a physical Pixel device, so I just tested it on an Pixel 3 emulator and it worked for me, but I know that camera2 API might not be working the same as it does on physical devices

[ibmmt]

const photo = await cameraRef.current.takePhoto(); let rotation=0; if ( isFrontCamera && (photo.orientation == 'landscape-left' || photo.orientation == 'landscape-right')) { rotation = 90; } const photoWithOrientation = await manipulateAsync(photo.path, [{ rotate: rotation }], { compress: 1, format: SaveFormat.JPEG, });

Temporary solution.

[JamesHemery]

@mrousavy Unfortunately, I don't have a physical Pixel device, so I just tested it on an Pixel 3 emulator and it worked for me, but I know that camera2 API might not be working the same as it does on physical devices

On Pixel 4a this solves the problem on the front camera, but creates a problem on the rear camera.

https://youtube.com/shorts/J0jY1ziqTT0

[JamesHemery]

This works with front/rear camera condition :

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
index 88c085f..2293210 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
@@ -65,13 +65,22 @@ suspend fun CameraView.takePhoto(optionsMap: ReadableMap): WritableMap {
   }
 }

-private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File) {
+private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File, cameraCharacteristics: CameraCharacteristics) {
   val byteBuffer = photo.image.planes[0].buffer
   if (photo.isMirrored) {
     val imageBytes = ByteArray(byteBuffer.remaining()).apply { byteBuffer.get(this) }
     val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
     val matrix = Matrix()
-    matrix.preScale(-1f, 1f)
+
+    val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT
+    if (isFront){
+      val sensorOrientation = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!.toFloat()
+      matrix.setRotate(sensorOrientation)
+      matrix.postScale(-1f, 1f, bitmap.getWidth() / 2f, bitmap.getHeight() / 2f);
+    } else {
+      matrix.preScale(-1f, 1f)
+    }
+
     val processedBitmap =
       Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
     FileOutputStream(file).use { stream ->
@@ -94,7 +103,7 @@ private suspend fun savePhotoToFile(
       // When the format is JPEG or DEPTH JPEG we can simply save the bytes as-is
       ImageFormat.JPEG, ImageFormat.DEPTH_JPEG -> {
         val file = createFile(context, ".jpg")
-        writePhotoToFile(photo, file)
+        writePhotoToFile(photo, file, cameraCharacteristics)
         return@withContext file.absolutePath
       }

It's not ideal, but it helps as a temp fix.

[isinuyk]

@JamesHemery I don't think
val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT

this condition is required. as you can see you already doing something inside of photo.isMirrored scope and if you take a look on isMirrored definition you will see

that this is exactly the same as you did in isFront.

[JamesHemery]

@JamesHemery I don't think val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT

this condition is required. as you can see you already doing something inside of photo.isMirrored scope and if you take a look on isMirrored definition you will see

that this is exactly the same as you did in isFront.

Wtf, Indeed. But it solves the problem of the video I sent earlier. Without this condition, the rear camera photo is turned upside down. There may be an isMirrored switch or wizardry somewhere.

I'll be digging deeper at the end of the day.

[CongDuongLe]

@JamesHemery I want to hear more of your opinions

[carlos-yoinks]

Wtf, Indeed. But it solves the problem of the video I sent earlier. Without this condition, the rear camera photo is turned upside down. There may be an isMirrored switch or wizardry somewhere.

It worked for me. I need only to take a photo on Android and IOS devices. I don't know about recording videos yet.

[kadinuguru]

@JamesHemery I don't think val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT this condition is required. as you can see you already doing something inside of photo.isMirrored scope and if you take a look on isMirrored definition you will see that this is exactly the same as you did in isFront.

Wtf, Indeed. But it solves the problem of the video I sent earlier. Without this condition, the rear camera photo is turned upside down. There may be an isMirrored switch or wizardry somewhere.

I'll be digging deeper at the end of the day.

I am facing a similar issue. I have installed the latest version of react-native-vision-camera and I manually updated the '/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt' file as suggested in the previous post. I have tried both with and without this change and I don't see a difference in the output image. Please correct me if I am not doing this right

Here is how my imaging screen looks like

and here is how the captured image looks

{"height": 3060, "isMirrored": false, "isRawPhoto": false, "orientation": "landscape-right", "path": "/data/user/0/com.xxxxxxxx/cache/mrousavy6198850539183917807.jpg", "width": 4080}

the end goal is to get an image that closely matches what is shown in the camera view while maintaining 4:3 (Width/Height) aspect ratio.

While we are waiting for the orientation implementation to be included as part of the future release is there a fix for this issue?

[hashhirr]

I've been investigating this bug for a while and found the root of the problem. So, the first thing that you need to know is that isMirroredvalue is strictly defined as true if the current device is a Front camera.

Then we are using this isMirroredvariable inside of the writePhotoToFilefunction after the picture was taken.

And there is the main thing. When we have isMirrored = true in that case we will create the Matrix() of our image and scale it to flip it back into the correct view. In the case of the back camera we skip this phase and just write the byte array of this file.

For some reason when we are creating this Matrix() it is rotating our image to the camera sensor orientation. For example, in my case, the front camera orientation is 270 and my resulting image was always rotated by 270 degrees. So, all I did was just not only flip the matrix horizontally as it was already there but additionally rotate the matrix for the value of the sensor orientation.

So, this is how my patch looks like at the end.

react-native-vision-camera+3.6.4.patch

I've tested it on Samsung A11, Motorola g72 - Android 12, and Pixel 3

@mrousavy what do you think? maybe it is worth adding this as a patch to next versions or should we wait for the global Orientation feature?

it worked for me in Xiaomi poco X3, Samsung A03 and pixel simulator...

[bglgwyng]

Try this patch script.

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt
index 7f2e8dd..5ffb34f 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/CameraSession.kt
@@ -324,6 +324,7 @@ class CameraSession(private val context: Context, private val cameraManager: Cam
         size.width,
         size.height,
         video.config.pixelFormat,
+        Orientation.fromRotationDegrees(characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION) ?: 0),
         isSelfie,
         video.config.enableFrameProcessor
       )
diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt
index 0b2b69b..8269fc0 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt
@@ -30,6 +30,7 @@ class VideoPipeline(
   val width: Int,
   val height: Int,
   val format: PixelFormat = PixelFormat.NATIVE,
+  private val orientation: Orientation,
   private val isMirrored: Boolean = false,
   enableFrameProcessor: Boolean = false
 ) : SurfaceTexture.OnFrameAvailableListener,
@@ -110,7 +111,7 @@ class VideoPipeline(
         val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener

         // TODO: Get correct orientation and isMirrored
-        val frame = Frame(image, image.timestamp, Orientation.PORTRAIT, isMirrored)
+        val frame = Frame(image, image.timestamp, orientation, isMirrored)
         frame.incrementRefCount()
         frameProcessor?.call(frame)

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/extensions/CameraDevice+createPhotoCaptureRequest.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/extensions/CameraDevice+createPhotoCaptureRequest.kt
index 0c425a8..5e66a7d 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/extensions/CameraDevice+createPhotoCaptureRequest.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/extensions/CameraDevice+createPhotoCaptureRequest.kt
@@ -51,7 +51,7 @@ fun CameraDevice.createPhotoCaptureRequest(
   }
   captureRequest.set(CaptureRequest.JPEG_QUALITY, jpegQuality.toByte())

-  captureRequest.set(CaptureRequest.JPEG_ORIENTATION, orientation.toDegrees())
+  captureRequest.set(CaptureRequest.JPEG_ORIENTATION, 0)

   // TODO: Use the same options as from the preview request. This is duplicate code!

You should rotate the photos explicitly with metadata.Orientation.

It's because the change captureRequest.set(CaptureRequest.JPEG_ORIENTATION, 0) in the patch script makes the photo unchanged as the sensor's rotation. Of course, it's inconvenient, but I found that Galaxy S21 doesn't respect the value of CaptureRequest.JPEG_ORIENTATION when the photo is from the front camera. Yes, it's crazy.

captureRequest.set(CaptureRequest.JPEG_ORIENTATION, 0) lets you rotate the photos on your own, but it guarantees consistency between the camera devices. You can use expo-image-manipulator the rotate the photos.

[mrousavy]

Interesting research! Sucks that we have to rotate photos afterwards, as this definitely reduces performance.

[ziga-hvalec]

This works with front/rear camera condition :

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
index 88c085f..2293210 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
@@ -65,13 +65,22 @@ suspend fun CameraView.takePhoto(optionsMap: ReadableMap): WritableMap {
   }
 }

-private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File) {
+private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File, cameraCharacteristics: CameraCharacteristics) {
   val byteBuffer = photo.image.planes[0].buffer
   if (photo.isMirrored) {
     val imageBytes = ByteArray(byteBuffer.remaining()).apply { byteBuffer.get(this) }
     val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
     val matrix = Matrix()
-    matrix.preScale(-1f, 1f)
+
+    val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT
+    if (isFront){
+      val sensorOrientation = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!.toFloat()
+      matrix.setRotate(sensorOrientation)
+      matrix.postScale(-1f, 1f, bitmap.getWidth() / 2f, bitmap.getHeight() / 2f);
+    } else {
+      matrix.preScale(-1f, 1f)
+    }
+
     val processedBitmap =
       Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
     FileOutputStream(file).use { stream ->
@@ -94,7 +103,7 @@ private suspend fun savePhotoToFile(
       // When the format is JPEG or DEPTH JPEG we can simply save the bytes as-is
       ImageFormat.JPEG, ImageFormat.DEPTH_JPEG -> {
         val file = createFile(context, ".jpg")
-        writePhotoToFile(photo, file)
+        writePhotoToFile(photo, file, cameraCharacteristics)
         return@withContext file.absolutePath
       }

It's not ideal, but it helps as a temp fix.

This one worked for me. Both back and front camera. Did anyone find any issues with this patch?

[mrousavy]

Also closing this as this is being tracked in https://github.com/mrousavy/react-native-vision-camera/issues/1891.

[yoann84]

This works with front/rear camera condition :

diff --git a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
index 88c085f..2293210 100644
--- a/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
+++ b/node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
@@ -65,13 +65,22 @@ suspend fun CameraView.takePhoto(optionsMap: ReadableMap): WritableMap {
   }
 }

-private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File) {
+private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File, cameraCharacteristics: CameraCharacteristics) {
   val byteBuffer = photo.image.planes[0].buffer
   if (photo.isMirrored) {
     val imageBytes = ByteArray(byteBuffer.remaining()).apply { byteBuffer.get(this) }
     val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
     val matrix = Matrix()
-    matrix.preScale(-1f, 1f)
+
+    val isFront = cameraCharacteristics.get(CameraCharacteristics.LENS_FACING) == CameraCharacteristics.LENS_FACING_FRONT
+    if (isFront){
+      val sensorOrientation = cameraCharacteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!.toFloat()
+      matrix.setRotate(sensorOrientation)
+      matrix.postScale(-1f, 1f, bitmap.getWidth() / 2f, bitmap.getHeight() / 2f);
+    } else {
+      matrix.preScale(-1f, 1f)
+    }
+
     val processedBitmap =
       Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
     FileOutputStream(file).use { stream ->
@@ -94,7 +103,7 @@ private suspend fun savePhotoToFile(
       // When the format is JPEG or DEPTH JPEG we can simply save the bytes as-is
       ImageFormat.JPEG, ImageFormat.DEPTH_JPEG -> {
         val file = createFile(context, ".jpg")
-        writePhotoToFile(photo, file)
+        writePhotoToFile(photo, file, cameraCharacteristics)
         return@withContext file.absolutePath
       }

It's not ideal, but it helps as a temp fix.

This one worked for me. Both back and front camera. Did anyone find any issues with this patch?

Yes we had devices with bad orientation with this patch. For example with this device : "zte grand x view 4"