Transparent video exported as "premultipied" renders incorrectly

[michelepanegrossi]

Describe the issue A clear and concise description of what the issue is.

It appears that portions of video frames where the alpha is very low (video with alpha packing) don't render properly as if they got 'filtered out'. This problem only seems to occur if the background is completely transparent. If there is something else in the video that partially covers the background, the issue occurs on the transparent portions of the frame but not the others (I guess because the alpha there is higher anyway). This is not the case if a dark UI element is used as background, such as a black Panel. It has to be in the video.

Your Setup (please complete the following information):

• Unity version: 2021.2.9f1
• AVPro Video version (number and edition (trial/core/ultra/enterprise)): 2.5.1 core edition
• Operating system version: Windows 10
• Device model: Desktop PC with Nvidia RTX 3080, AMD Ryzen 5900x, 32GB RAM
• Video specs (resolution, frame-rate, codec, file size):

4096 * 1652, 60 FPS, H264 MP4, some 100MB some 8MB, alpha packing left-right.

To Reproduce

1. Install Unity HDRP. Open new scene, set resolution to 2160 x 3840.
2. Add Display IMGUI component to empty GameObject.
3. Add Media Player to empty GameObject, in Windows options select only hardware decoding and nothing else. All other settings default. Set video to auto load and auto play.
4. Setup media reference with video in streaming asset folder, transparent video, left right packing.
5. Enter play mode

In the attached video, only the animated text doesn't render properly. Other portions of the frame which have high alpha values render fine.

Logs If applicable, add error logs to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem.

Videos If applicable, add a copy of your video or the URL https://user-images.githubusercontent.com/45626940/172477018-f7cd9e08-2bc4-4818-a3b1-7a9e64cdb077.mp4

Please DO NOT LINK / ATTACH YOUR PROJECT FILES HERE

Instead email the link to us unitysupport@renderheads.com

[AndrewRH]

Hi,

I investigated this issue. I believe the problem is with the way the video has been authored.

The text on the left side seems to already have the alpha applied to it. I would expect the text on the left to be a solid colour (white), and the text on the right (alpha channel) to have greyscale animation applied to it.

I hope this helps to resolve your issue?

Thanks,

[AndrewRH]

Upon further inspection, I suspect that you exporting your transparent video using "Premultiplied alpha" instead of "Straight alpha". If you're using After Effects you can switch to "Straight" and it should resolve this issue.

Does that help?

Thanks,

[michelepanegrossi]

Thanks @AndrewRH I think I have authored a video as you suggested in your first message but it doesn't seem to make any difference.

See the video here https://user-images.githubusercontent.com/45626940/172636658-fc556a97-7c51-4d8b-8930-de62abe6eb30.mp4

Do you think I should export this new video with straight alpha or the previous one I sent?

Are you suggesting I use mov files instead of mp4? What codec would work on windows with a mov file?

Many thanks

[michelepanegrossi]

I tried a different approach. I exported 2x MOV ProRes4444 videos, one with premultiplied alpha and one with straight alpha.

Then I have converted to H264 with alpha packing using the ffmpeg commands provided in the Encoding notes:

ffmpeg -i input.mov -c:v libx264 -pix_fmt yuv420p -vf "split [a], pad=iw*2:ih [b], [a] alphaextract, [b] overlay=w" -y output.mp4

EDIT: The conversion worked, but the result is not so different. The first couple of seconds of the text animation are still very hard to see although a little visible now.

Here are the video outputs from ffmpeg: https://user-images.githubusercontent.com/45626940/172711423-84724f09-952f-4e03-a6f2-dcef77c2cd09.mp4 https://user-images.githubusercontent.com/45626940/172711392-6c0e53c0-3eb5-4f76-be94-7984592cf10a.mp4

They give exactly the same result in Unity, despite looking different.

The only thing that seems to work is to increase the alpha, like this https://user-images.githubusercontent.com/45626940/172711690-5ca76aef-2961-4330-a1ea-d628749d901b.mp4

Do you have any thoughts?

[AndrewRH]

Do you have a reference of how you expect it to look?

[michelepanegrossi]

I think it should look like this video. This is an H264 mp4 export form the straight alpha ProRes mov file. https://user-images.githubusercontent.com/45626940/172731471-d3b23e3b-a000-411e-a837-ed209271837a.mp4

I did more checking in Unity and it would seem that the ffmpeg version I got from the straight alpha version is actually pretty close (it's difficult to judge as I cannot play the ProRes file in Unity on windows).

I think you were probably right on your second comment.

Many thanks for your help with this matter.

Do you think you could explain why it is necessary to export the video with straight alpha?

[AndrewRH]

Well, our shaders expect the RGB channel to be 'straight' and not premultipied with the alpha channel, so if it's not in this format then the resuilts won't be as expected. We can change the shader to support premultiplied mode but we decided just to pick one method instead of trying to support both. Perhaps in the future we will support both and allow the user to select the mode to use. Unfortunately this information is not stored within a video so there is no way for us to detect the format automatically.

Perhaps another reason for the difference is compression artifacts. I notice that the text that animates in has a lot of fine detail. These details can lose due clarity if the compression level is set too high, or there is a limit on the bitrate. So you may want to consider increasing the quality (decreasing compression) and/or increasing the bitrate.

Glad that it's working for you now. I will close this issue.