MediaPlayer.ExtractFrame returns only red channel on iOS in Y'CbCr mode

[Murcho]

Describe the bug When using MediaPlayer.ExtractFrame on iOS, the returned texture only seems to include a red channel. The exact same code extracts correct screenshots under Android, Windows Editor and MacOS Editor.

Your Setup (please complete the following information):

• Unity version: 2019.1.8
• AVPro Video version: 1.10.0
• Operating system version: iOS 12.3.1
• Device model: iPad Mini 2
• Video specs (resolution, frame-rate, codec, file size): 1080x1920, .mov container (recorded from native camera on iOS device). Example movie file is provided in attached zip.

To Reproduce

1. Load a movie into a Media Player
2. Call "ExtractFrame" on the media player in an iOS build
3. Check the provided Texture, it is very red.

Logs If applicable, add error logs to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem. In this zip is the image that was extracted on iOS, and the movie it came from. ExampleMovieAndThubnail.zip

Videos If applicable, add a copy of your video or the URL

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

[MorrisRH]

The iOS media player by default uses a bi-planar texture format for movie playback which is currently unsupported by the extract frames feature. However you can work around this by unchecking “Use YpCbCr420” in the iOS specific section of the media player settings. This puts the player back to using standard RGBA textures which will work with the extract frames feature.

[Murcho]

Just tested this work around and can confirm it works. Thanks for the speedy response.

You might want to leave this as a bug until the bi-planar texture format is supported by the Extract Frames feature.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

[roger-o3h]

Thanks for keeping this on the board for 2.0. Just cheering you on and letting you know there's still interest in this!

[roger-o3h]

As a workaround, you can manually use one of the display shaders to transform the two luma/chroma textures to a readable RGB texture.

if (MediaPlayer.GetPlatform() == Platform.iOS && mediaPlayer.PlatformOptionsIOS.useYpCbCr420Textures)
{
    var renderMaterial = new Material(Shader.Find("AVProVideo/Unlit/Opaque (texture+color+fog+stereo support)"));
    renderMaterial.EnableKeyword("USE_YPCBCR");

    // setup material
    var mat = mediaPlayer.TextureProducer.GetYpCbCrTransform();
    Debug.Log($"YpCbCr tx {mat}");
    renderMaterial.SetMatrix("_YpCbCrTransform", mat);

    int textureCount = mediaPlayer.TextureProducer.GetTextureCount();
    string[] textureProps = {"_MainTex", "_ChromaTex"};
    for (int i = 0; i < Math.Min(textureCount, textureProps.Length); i++)
    {
        if (mediaPlayer.TextureProducer.GetTexture(i) is Texture2D tex)
        {
            renderMaterial.SetTexture(textureProps[i], tex);
        }
    }

    Texture srcTex = mediaPlayer.TextureProducer.GetTexture(0);
    displayTexture = new Texture2D(srcTex.width, srcTex.height, TextureFormat.RGB24, false);
    RenderTexture rt = RenderTexture.GetTemporary(displayTexture.width, displayTexture.height, 0, RenderTextureFormat.Default);
    RenderTexture.active = rt;
    GL.Clear(true, true, Color.clear);
    Graphics.Blit(null, rt, renderMaterial);
    displayTexture.ReadPixels(displayTexture.GetFullRect(), 0, 0);
    displayTexture.Apply();
    RenderTexture.ReleaseTemporary(rt);
}

[AndrewRH]

This will be fixed when the texture resolver feature is implemented and the ExtractTexture method refactored This isn't a big deal at the moment as it only happens when using the YCbCr mode on iOS which isn't the default.

[thesanketkale]

Hey Guys, I am getting this red channel issue when I am trying to playback a local HEVC video from the iPhone's gallery in a Unity app on a RenderTexture. Whenever the video is played, the whole screen turns red on the iPhone.

Actually, I needed to dynamically fit the video inside a RenderTexture on a 3D Quad preserving the aspect ratio of the video selected by the user. I am doing this by writing the extracted frame of the video onto the Render Texture using a custom shader.

I tried changing the texture format on the iOS platform like below, but it didn't help. I did this by adding the following code: mediaPlayer.PlatformOptionsIOS.textureFormat = MediaPlayer.OptionsApple.TextureFormat.BGRA;

I also tried @roger-o3h's workaround, but it is throwing compile errors near the line displayTexture.ReadPixels(displayTexture.GetFullRect(), 0, 0);

It says Texture2D does not contain a definition of GetFullRect().

I am using AvPro version 2.8.4 in Unity Editor version 2022.3.4f1 with the iOS target platform and I tested the build on an iPhone 14 Pro Max.

How can I solve this? Is there anything different to be done in the latest versions to solve this?

[MorrisRH]

Hey @thesanketkale, I'm afraid you're issue is different to this one. This issue has everything tinted red as ExtractFrame is only using the luma channel, not the solid red that you're seeing.

We'll create a new issue for adding scaling modes to the ResolveToRenderTexture component which is what I think would fix your problem.

[thesanketkale]

Great, the scaling mode feature in ResolveToRenderTexture will be of great help. Thanks.

Besides this, is there any temporary workaround for me to reliably extract frames from the MediaPlayer other than the ExtractFrame() method and apply scaling from my side till this feature comes out? I tried the TextureProducer but that texture does not come with orientation correction. The red screen comes only while playing an HEVC video when using the ExtractFrame method. Is this an issue with using ExtractFrame?

Edit: I have raised a new issue here. Let's continue it there. Thanks.