Rotation degrees of Frame object is incorrect for Frame Processor

[jllarraz]

Hi,

I am facing a similar issue that the one described in #253. But in this case the problem is that when the phone is on landscape the rotation of the image says 270 but in reality it's 90. when it's on portrait it says 0 and that's ok. So I think that there is a problem with the image rotation that it's not taking in consideration the orientation of the sensor in the phone.

Note: the device used is a Samsung S9 and the back camera

[jllarraz]

Sorry I forgot to mention that I am using the 2.7.0 version

[jllarraz]

I had to use a custom function to get the right rotation, in case that someone finds himself in the same situation

fun getRotation(context: Context, facingCamera:Int =CameraCharacteristics.LENS_FACING_BACK):Int{
        val manager = context.getSystemService(Context.CAMERA_SERVICE) as android.hardware.camera2.CameraManager
        try {
            for (cameraId in manager.getCameraIdList()) {
                val characteristics = manager.getCameraCharacteristics(cameraId)

                // We don't use a front facing camera in this sample.
                val facing = characteristics.get(CameraCharacteristics.LENS_FACING)
                if (facing != null && facing != facingCamera) {
                    continue
                }

                val mSensorOrientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!
                val rotation = (context.getSystemService(Context.WINDOW_SERVICE) as WindowManager).defaultDisplay.rotation
                var degrees = 0
                when (rotation) {
                    Surface.ROTATION_0 -> degrees = 0
                    Surface.ROTATION_90 -> degrees = 90
                    Surface.ROTATION_180 -> degrees = 180
                    Surface.ROTATION_270 -> degrees = 270
                }
                var result: Int
                if (facing == CameraCharacteristics.LENS_FACING_FRONT) {
                    result = (mSensorOrientation + degrees) % 360
                    result = (360 - result) % 360  // compensate the mirror
                } else {  // back-facing
                    result = (mSensorOrientation - degrees + 360) % 360
                }
                return result
            }
        } catch (e:Exception){
        }
        return 0
    }

[Diolor]

Hey, is this affecting frames coming from frame processor only or when you .takePicture() as well?

[jllarraz]

I am only using the preview, so I can't say if take pictures has the same issue

[jllarraz]

A little improvement over my previous method, as I had an error with the front camera rotation

fun getRotation(context: Context, lensPosition: LensPosition = LensPosition.Back):Int{

        var facingCamera=0
        when(lensPosition){
            LensPosition.Front->{
                facingCamera = CameraCharacteristics.LENS_FACING_FRONT
            }
            LensPosition.Back->{
                facingCamera = CameraCharacteristics.LENS_FACING_BACK
            }
            LensPosition.External->{
                facingCamera = CameraCharacteristics.LENS_FACING_EXTERNAL
            }
        }

        val manager = context.getSystemService(Context.CAMERA_SERVICE) as android.hardware.camera2.CameraManager
        try {
            for (cameraId in manager.getCameraIdList()) {
                val characteristics = manager.getCameraCharacteristics(cameraId)
                val facing = characteristics.get(CameraCharacteristics.LENS_FACING)
                if (facing != null && facing != facingCamera) {
                    continue
                }

                val mSensorOrientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!
                val rotation = (context.getSystemService(Context.WINDOW_SERVICE) as WindowManager).defaultDisplay.rotation
                var degrees = 0
                when (rotation) {
                    Surface.ROTATION_0 -> degrees = 0
                    Surface.ROTATION_90 -> degrees = 90
                    Surface.ROTATION_180 -> degrees = 180
                    Surface.ROTATION_270 -> degrees = 270
                }
                var result: Int
                if (facing == CameraCharacteristics.LENS_FACING_FRONT) {
                    result = (mSensorOrientation + degrees - 360) % 360
                    result = (360 + result) % 360  
                } else {  // back-facing
                    result = (mSensorOrientation - degrees + 360) % 360
                }
                return result
            }
        } catch (e:Exception){
        }
        return 0
    }

[lsuski]

I confirm, for Nexus 5X it should be 270 degrees but is 90

[tomblenz]

Google's vision kit has a method called getRotationCompensation to retrieve the degrees by which a frame must be rotated to be in the correct orientation - https://github.com/firebase/snippets-android/blob/e1769c791903a8b417cfe761909782aca718c4f8/mlkit/app/src/main/java/com/google/firebase/example/mlkit/kotlin/VisionImage.kt

// On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees.

The 'why' isn't mentioned, but the method is useful elsewhere, eg. OP's problem. Not sure if a 3rd party library should try to fix this...

[YodaEmbedding]

Here's my version:

private val rotationToDegrees = mapOf(
    ROTATION_0 to 0,
    ROTATION_90 to 90,
    ROTATION_180 to 180,
    ROTATION_270 to 270
)

@Throws(CameraAccessException::class)
fun getRotationCompensation(context: Context, facing: Int): Int {
    val windowManager = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager
    val deviceRotation = rotationToDegrees.getValue(windowManager.defaultDisplay.rotation)

    val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
    val characteristics = cameraManager.cameraIdList
        .map { cameraManager.getCameraCharacteristics(it) }
        .first { it.get(CameraCharacteristics.LENS_FACING) == facing }
    val sensorOrientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!

    val polarity = if (facing == CameraCharacteristics.LENS_FACING_FRONT) 1 else -1
    return floorMod(sensorOrientation + polarity * deviceRotation, 360)
}

fun lensPositionToLensFacing(lensPosition: LensPosition): Int {
    return when (lensPosition) {
        LensPosition.Front -> CameraCharacteristics.LENS_FACING_FRONT
        LensPosition.Back -> CameraCharacteristics.LENS_FACING_BACK
        LensPosition.External -> CameraCharacteristics.LENS_FACING_EXTERNAL
    }
}

Example usage:

// MainActivity

val facing = lensPositionToLensFacing(CameraCharacteristics.LENS_FACING_BACK)
val degrees = getRotationCompensation(this, facing)

If you have the camera ID already, you can use it directly instead of specifying facing:

@Throws(CameraAccessException::class)
fun getRotationCompensation(context: Context, cameraId: String): Int {
    val windowManager = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager
    val deviceRotation = rotationToDegrees.getValue(windowManager.defaultDisplay.rotation)

    val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
    val characteristics = cameraManager.getCameraCharacteristics(cameraId)
    val facing = characteristics.get(CameraCharacteristics.LENS_FACING)
    val sensorOrientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION)!!

    val polarity = if (facing == CameraCharacteristics.LENS_FACING_FRONT) 1 else -1
    return floorMod(sensorOrientation + polarity * deviceRotation, 360)
}