XRCpuImage Rotation and Clipping Issue Affecting AI Model Application

[AntoineChauviere]

Hello,

I am trying to apply my AI model to an XRCpuImage. It works very well, but I've noticed that the XRCpuImage is rotated by 90 degrees and seems to be clipped. This results in the image I see on my phone not matching the image to which I am applying the AI (XRCpuImage).

I would like to know how to obtain an XRCpuImage that matches what is displayed on the screen. This way, I can correctly perform conversions (such as rescaling, RGB24, etc.) on the correct image before applying the AI.

Here is a snippet of my code where the issue needs to be addressed:

public void ProcessCameraImageAsync(XRCpuImage image)
{
  var conversionParams = new XRCpuImage.ConversionParams
  {
      // Configure the conversion
      inputRect = new RectInt(0, 0, image.width, image.height),
      outputDimensions = new Vector2Int(320, 256),
      outputFormat = TextureFormat.RGB24,
      transformation = XRCpuImage.Transformation.None
  };

  // Texture size
  int size = image.GetConvertedDataSize(conversionParams);

  // Allocate a buffer for the image data
  var buffer = new NativeArray<byte>(size, Allocator.Persistent);

  // Convert the image
  image.Convert(conversionParams, buffer);

  // Update the displayed texture with the converted data
  displayTexture.LoadRawTextureData(buffer);
  displayTexture.Apply();

  // Access pixel data
  byte[] pixelData = buffer.ToArray();
 };

I need to apply the necessary changes to ensure that the XRCpuImage matches what is displayed on the screen and correct for any clipping issues before proceeding with the conversion and application of the AI model.

Any guidance or suggestions on how to resolve this issue would be greatly appreciated.

Thank you!

[tdmowrer]

The comment in the Depth Images sample explains why this is, and provides some pointers on how to convert between the raw and displayed images.