Premultiplied Alpha Question

[petzo]

Hi,

I'm having issues understanding how alpha works in Caspar. I think I get all the straight/premultiplied maths, but things doesn't look correct nevertheless. I made the following test:

1. I make a solid green layer (layer 1)
2. I load the test pattern video from the preinstalled ones in caspar (the 720p6000) in particular. (layer 2)
3. I use a screen consumer for the merit of the test
4. I use the other default config settings for everything

I get this:

I expected to get this:

Shouldn't Caspar mixer do a better job of mixing those? Or maybe I'm doing something wrong?

(I guess this question is mostly suited for the forum, but I can't register there as there is an issue with the reCaptcha in the sign up - keeps telling me I've entered wrong letters...)

[jesperstarkar]

All internal processing in CasparCG is premultiplied, so you want your sources to be premult.

You have an option to "un-premultiply" in the final stage in the MIXER before the CONSUMER to have straight alpha output.

Am I right @HellGore ? I might be wrong...

[petzo]

Thanks for the info, but I want to make something clear here. This final result is without alpha. That is because the green base layer is solid. The second layer which I load is with premultiplied alpha (from the included demo files). So I cant "un-premultiply" as this final result is without alpha at all.

[jesperstarkar]

OK, so either the keying in CasparCG is broken, or the sample file has incorrect alpha values. You sure the file is premultiplied?

Ping @hummelstrand

[petzo]

Yes, I'm sure. I use the provided test patterns. Just re-downloaded them from here: https://github.com/CasparCG/Server/blob/2.1.0/deploy/general/server/media/TestPatterns/720p6000_Test.mov https://github.com/CasparCG/Server/blob/2.1.0/deploy/general/server/media/TestPatterns/720p6000_Test_a.mov I use the stable 2.0.7.

[petzo]

Just tested it in latest 2.1.0 build - same results. It looks like the internal mixer is expecting straight alpha, which is against everything I've read in the wiki or forums. As it is a very easy to set up this test, can anyone else reproduce the issue?

[HellGore]

If you look at our discussion here https://github.com/CasparCG/Server/pull/547 I concluded that the test patterns must be wrong somehow. Or at least I don't understand how to interpret them.

[5opr4ni]

Hi

With normal videofiles/images (the normal way) that has embedded alpha, it works as it should (premult). I have made a test.

It's when you use a file with an XXXXX-A.mov that works as an mask for a video XXXXX.mov (-A.mov) this happens. We will have a look at that. Why do we have this way of doing things? This is a way of getting very small files with alpha, we don't use them at all, but it is good that you pointed this out. This artifact also happens when setting a layer as a mask for another layer. (Mixer mask) /olle soprani

[5opr4ni]

Hi, one last thing if you want to work with masking version (one file for fill and another (-a) as a mask. Make the fill straight, and it will look god. I will do a test to verify.

/olle

[5opr4ni]

OK!

As I wrote, if you have a straight fill it works as it should. The sample files provided is not as they should be. We will have a discussion in the team about if we will make this a parameter. To have the fill straight or premultiplied when masked by other source. This is how the file you tested with if we use the maskfile (-a) as a mask to a completely white layer.

Here you have two samples of the test videos with straight fill instead. Archive.zip

[petzo]

Thanks @5opr4ni

I've reached the same conclusion - that if you play a video pair of key + fill, it needs to be prepared as straight alpha, and not as premultiplied alpha. And this is what is confusing, because everywhere it is discussed that alpha should be premultiplied as a rule. Example in the wiki here says: http://casparcg.com/wiki/Content_/_Media#Key_.2F_Alpha_Channel

The alpha channel in videos played by the FFmpeg Producer should be premultiplied so you can play content from any other producer on top and have everything correctly composited.

So my suggestion would be to make the keying to expect premultiplied as default and not straight as now.

As I'm getting familiar with the software I thought this video pair of two files for fill + key is the preferred way to do it. That is because all the supplied demo files are done this way. And when they don't work out of the box this is really confusing. Also from your comment I get the feeling that this is not the preferred way to play video with alpha? What is the preferred way? Maybe .MOV with ProRes in a single file?

[petzo]

I've done some additional test with the provided pattern files and some of my own. I have the following conclusions:

1. :white_check_mark: Video with an embedded alpha channel, fill is premultiplied (expected)
2. :no_entry_sign: Video with an embedded alpha channel, fill is straight (not premultiplied) (expected)
3. :no_entry_sign: Video with two separate files - key + fill, fill is premultiplied (NOT expected)
4. :white_check_mark: Video with two separate files - key + fill, fill is straight (NOT expected)
5. :no_entry_sign: TGA file - doesn't work as expected even that it is premultiplied by default (NOT expected)
6. :white_check_mark: PNG file - work as expected even that PNGs are straight by default (expected)

Legend: :white_check_mark: - it works :no_entry_sign: - doesn't work

Some strange details from the TGA test: When I load the TGA on a black background it displays this (like it is straight):

When I put a green solid behind the TGA, this happens:

I don't really know why this happens...

So to sum it up - I think Caspar should change (fix) its behavior regarding points 3, 4 & 5 from my list above.

[5opr4ni]

Hi

The preferred way is NOT to have two files. This is the way it was done when you used to have beta machines (tape). The preferred way is to have one file with fill and key. Actually that was the first reason why we built Caspar. The reason we have supplied it in this manner is size, so you don't have to wait forever when downloading Caspar. Normally when you do cg and graphics you don't use a format that has so high compression as h.264. The best is to use QuickTime animation or QuickTime tga or tif. Prores with alpha works if you have a Real power-horse, it is really cpu intense to decode. For over 20 years as a designer, developer and operator I have never used two separate files if I can have one.

You often get straight fill graphics (sadly) from post houses when you are on a production and to have the possibility to handle those without having to go trough aftereffects are a lifesaver.

Regarding the image you have tested with. There are different things to look for to determine how a Visionmixer in a controllrom applies the DSK and this image is made for that, you should not use it as you do.

We can not provide a course in how to use graphics in broadcast, there are a lot of resources on internet to learn about this. Sorry!

/Olle

[petzo]

@5opr4ni

Thank you for your comments regarding best practices to use Caspar! Still, I think it is good idea to address the issues listed above, even if they are not the most common scenarios to work with.

Best regards.

[5opr4ni]

I updated the last answer.

[petzo]

I see, so everything listed is expected behavior. I understand. Well, lets hope than that this list is of some help for someone else spending hours trying to understand how Caspar works as I did :)

Best regards.

[5opr4ni]

Everything is expected behavior, but we will change the supplied test-files so confusion not arises and add a switch so that you can switch between straight and premultiplied when you work with mask files. I hope you learned a lot about how we work. I actually spent over one hour to help you with a software that I have made available for free and for you to use without spending a buck. Thanks petzo.

/olle

[petzo]

And I'm very thankful because of that. I really appreciate it! Thank you once again for your time and that of everyone in the team. I was already looking into the code actually to try to contribute with whatever I can. I'm currently experimenting with WebGL in the HTML Producer, and trying to see what can be improved there.

Thanks again and greeting from Bulgaria! :)

[jesperstarkar]

That list of yours is really good @petzo, I guess you have spent some hours researching and compiling that one too. I second your opinions on what's considered expected and not expected behaviours.

What @5opr4ni suggests, updating the sample files and perhaps adding another flag switching the keyer in the producer, are good solutions that would have avoided this confusion.

[dotarmin]

Hii,

(I guess this question is mostly suited for the forum, but I can't register there as there is an issue with the reCaptcha in the sign up - keeps telling me I've entered wrong letters...)

I just created an test account on the forum and it worked. If you still experience any issues with the forum, please feel free to send me an email and I will personally help you.

Thanks for using CasparCG!

/Armin

[petzo]

Thank you @dotarmin - I was able to register. Now I'm waiting for the admin approval. I consider my questions answered so I close this issue and will post my other questions in the forum :)

[dotarmin]

Thank you @dotarmin - I was able to register. Now I'm waiting for the admin approval. I consider my questions answered so I close this issue and will post my other questions in the forum :)

I have now activated your account. If you find any bugs, have feature request or just suggestions for improvements, feel free to create an issue here on GitHub.

Best regards! /Armin