Alpha support

[ThomasKliszowski]

Hello 👋

I tried to find informations about alpha support in the documentation, but I couldn't find anything talking about that. Can you tell me if alpha is supported with vp8 / vp9 encoding?

I made a wasm implementation of libwebm to mux a .webm file from the WebCodecs video chunks, it works well but the alpha channel is ignored during the process.

Thank you 🙏

[padenot]

Thomas, this is the home of the specification, that is heavily in flux and not really stable.

You're right there is not a lot of info about pixel format (which is one of the things that are really needed, alongside color spaces). For now, YUV420 is the only supported format. Let's turn this into an issue about supported formats.

I very much expect that to be supported when the spec and implementations are ready, though.

[ThomasKliszowski]

Thanks @padenot for your quick reply.

That's clear, this limitation makes more sense to me now:

Constructing a VideoFrame from an ImageBitmap produces an I420 frame. We expect this to change to RGBA or an opaque format in the future.

[chcunningham]

Support for alpha is definitely in the works! Since this issue is now focused on pixel formats, I'll discuss the full scope of alpha proposals in a separate issue.

For pixel formats specifically, my vote is to use esatblished fourcc's. https://www.fourcc.org/yuv.php https://www.fourcc.org/rgb.php

Some of these describe alpha (e.g. RGBA), but this isn't the case for planar formats (which may also have alpha). Said another way, we don't see an I420A in the YUV list.

Rather than minting new "A" suffixed planar fourcc's, I vote to just add a hasAlpha bool on the VideoFrame. The semantics would be that, four formats with N non-alpha planes, they have N+1 planes when hasAlpha is true and the last plane (highest index) is the alpha plane.

[chcunningham]

Also, a note on naming of things like RGB vs BGR etc...

In Chrome's internal format enum, this has been a bit messy. ImageData.data is the one case we know of that has a web-exposed RGBA, and the order of the letters reflects the order of the bytes when read from the uint8 array (i.e. byte[0] gives you R). We should follow that model with our naming. There will be cases (e.g. android) where R does not actually come first, in which case we should signal this by using a different format string where letters are again ordered to be consistent with the byte order.

[chcunningham]

Triage note: marking extension, as adding new formats is purely additive.

Having said that, Chrome is ahead of the spec here and I will prioritize minting a few obvious formats (RGB, NV12, ...) ASAP.

[padenot]

This may be obvious, but needs to be specified. In general, no format is specified in the specification in today's Editor's draft, and this is blocking.

[padenot]

This covers formats others than YUV420 as said by https://github.com/w3c/webcodecs/issues/200#issuecomment-839450657, which is set to be defined in https://github.com/w3c/webcodecs/issues/165.

[Djuffin]

Now we've got pixel formats with alpha and alphaSideData in EncodedVideoChunkMetadata.

But it seems that VideoDecoder doesn't have a way to decode alpha

[Djuffin]

More context: https://github.com/w3c/webcodecs/issues/377

[Djuffin]

we should look if we can combine decoding for SVC layers, Dolby, and alpha layers

[nbutmickey]

Now we've got pixel formats with alpha and alphaSideData in EncodedVideoChunkMetadata.

But it seems that VideoDecoder doesn't have a way to decode alpha

any methods to decode alpha now ? we just need it

[chrisn]

Discussed in Media WG meeting, 10 October 2023 minutes