Received camera Image through CPU is returned rotated.

[PanMig]

Hello all,

I am trying to get the camera image and use it for some processing. The only problem is that while I successfully get the camera image when I apply it to a raw image to visualize it, the image is rotated on the Z axis by 90 degrees.

The placed raw image on the canvas is not rotated (0,0,0) and the target platform is android. Although visualizing the image is for debugging purposes my outer goal is to use it in a computer vision algorithm, that's why I need to check the orientation of the texture that I am registering to the algorithm.

I post the code for getting the latest camera image. The function is called after the user presses a button.

public unsafe void GetCameraImage()
    {
        XRCameraImage image;
        if (!cameraManager.TryGetLatestImage(out image))
            return;

        var conversionParams = new XRCameraImageConversionParams
        {
            // Get the entire image
            inputRect = new RectInt(0, 0, image.width, image.height),

            // Downsample by 2
            outputDimensions = new Vector2Int(image.width / 2, image.height / 2),

            // Choose RGBA format
            outputFormat = TextureFormat.RGBA32,

            // Flip across the vertical axis (mirror image)
            transformation = CameraImageTransformation.MirrorY
        };

        // See how many bytes we need to store the final image.
        int size = image.GetConvertedDataSize(conversionParams);

        // Allocate a buffer to store the image
        var buffer = new NativeArray<byte>(size, Allocator.Temp);

        // Extract the image data
        image.Convert(conversionParams, new IntPtr(buffer.GetUnsafePtr()), buffer.Length);

        // The image was converted to RGBA32 format and written into the provided buffer
        // so we can dispose of the CameraImage. We must do this or it will leak resources.
        image.Dispose();

        // At this point, we could process the image, pass it to a computer vision algorithm, etc.
        // In this example, we'll just apply it to a texture to visualize it.

        // We've got the data; let's put it into a texture so we can visualize it.
        m_Texture = new Texture2D(
            conversionParams.outputDimensions.x,
            conversionParams.outputDimensions.y,
            conversionParams.outputFormat,
            false);

        m_Texture.LoadRawTextureData(buffer);
        m_Texture.Apply();

        // Done with our temporary data
        buffer.Dispose();
    }

[vincentfretin]

Hi, you should reuse the NativeArray from the Texture2D to avoid a copy, LoadRawTextureData is doing a copy of buffer to the native array of the Texture2D. Look at CameraTextImage.cs in arfoundation-samples, it's using var rawTextureData = m_Texture.GetRawTextureData<byte>();

For the orientation of the device, you can get it with Screen.orientation. Here is some code of mine to rotate a matrix to account for the device orientation:

// on the class
static readonly Matrix4x4 invertZM = Matrix4x4.TRS (Vector3.zero, Quaternion.identity, new Vector3 (1, 1, -1));

// somewhere in the Update
                    Matrix4x4 localTransform = new Matrix4x4(.....some matrix you got from opencv, coordinates in the camera space, from the cpu image (x coordinates of the opencv detection Vector2d was mirrored though))
                        );
                    Matrix4x4 ARM = localTransform * invertZM;
                    float rotZ = 0;
                    switch (Screen.orientation) {
                        case ScreenOrientation.Portrait:
                            rotZ = 90;
                            break;
                        case ScreenOrientation.LandscapeLeft:
                            rotZ = 180;
                            break;
                        case ScreenOrientation.LandscapeRight:
                            rotZ = 0;
                            break;
                        case ScreenOrientation.PortraitUpsideDown:
                            rotZ = -90;
                            break;
                    }
                    Quaternion rotation = Quaternion.Euler(0, 0, rotZ);
                    Matrix4x4 m = Matrix4x4.Rotate(rotation);
                    ARM = m * ARM;
                    Matrix4x4 worldTransform = m_CameraManager.transform.localToWorldMatrix * ARM;
// https://forum.unity.com/threads/how-to-assign-matrix4x4-to-transform.121966/#post-819280
                    m_TriAxes.transform.FromMatrix(worldTransform);

[clifflinmao]

Can you the author anwser the question above?