search

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.
https://w3c.github.io/webcodecs/
Other
978 stars 136 forks source link

Define codec registry for EncodedAudioChunk and EncodedVideoChunk byte representations #155

Closed manifest closed 2 years ago

manifest commented 3 years ago

I'd like to encode audio and video frames in the browser, then process the chunks in an external application (e.g. a backend) and finally return them back to the browser. I assume that EncodedAudioChunk and VideoAudioChunk byte representations have to be stable because of the requirement of browser interoperability.

Where can I find information about EncodedAudioChunk and VideoAudioChunk byte representations? In particular, I'm interested in opus and vp8 codecs at the moment.

sandersdan commented 3 years ago

We expect this mapping to be described in a registry, the start of which is in #149.

More directly, each chunk is a complete sample and you can refer to the individual codec specifications for more details. Sample boundaries should match what you would expect in an MP4 containerization (eg. reference frames should be bundled into superframes). There is some ambiguity when SVC is involved that we are still analyzing.

manifest commented 3 years ago

Ok, I think that answers my question. Thank you.

sandersdan commented 3 years ago

@chcunningham I don't see any reference in #149 to packetization rules. There is a bit of text in the bitstream format part of the AVC entry, but I don't think it covers all of the questions we have received. (eg. "can I put SPS/PPS in an AVC format stream anyway?" and "should I send a buffer with just SPS/PPS for Annex B?")

chcunningham commented 3 years ago

I agree, we should add more clarity. Do you think codec registry is the way? It implies we have to create a registration for every codec (not otherwise our plan). It would simplify if we could find some way to describe this more generically in the main document.

sandersdan commented 3 years ago

So far I we've always chosen the rule of 'do it the way that it is spec'd for MP4 containerization', but we may want to relax those requirements in some cases. Every codec already has to be in the registry to give it a codec string, so it seems the logical place.

Note that even if we just reference the MP4 containerization specs, there are inevitable points of divergence. For example:

FYI the AVC-in-MP4 spec (ISO/IEC 14496-15) devotes 4 pages to defining "sample" (WebCodecs chunk) and "configuration" (WebCodecs description).

padenot commented 3 years ago

(There is probably an OGV spec we can point to for that.)

https://www.theora.org/doc/Theora.pdf#page=159, Appendix A, it's not very long as well.

chcunningham commented 3 years ago

Triage note: marked 'editorial', as I believe we have (codec-specific) consensus on what a chunk is, but the registry should be updated to describe that.

padenot commented 3 years ago

Per https://github.com/w3c/webcodecs/issues/155#issuecomment-805132865

chcunningham commented 2 years ago

Registry now defined. All codecs registered.