search

w3c / webcodecs

WebCodecs is a flexible web API for encoding and decoding audio and video.
https://w3c.github.io/webcodecs/
Other
985 stars 136 forks source link

Support for AV1 switch frames #747

Open kalradivyanshu opened 10 months ago

kalradivyanshu commented 10 months ago

AV1 allows use of switch frames to drop to lower resolution, this could be great while the support for spatial SVC is still not there. From the AV1 spec:

Switch Frame An inter frame that can be used as a point to switch between sequences. Switch frames overwrite all the reference frames without forcing the use of intra coding. The intention is to allow a streaming use case where videos can be encoded in small chunks (say of 1 second duration), each starting with a switch frame. If the available bandwidth drops, the server can start sending chunks from a lower bitrate encoding instead. When this happens the inter prediction uses the existing higher quality reference frames to decode the switch frame. This approach allows a bitrate switch without the cost of a full key frame.

Is there plans to add support for this in the AV1 encoder? Is it supported in the decoder?

sandersdan commented 10 months ago

My understanding is that switch frames should "just work" in decoders, but I have not tested.

How would you want to request switch frames from an encoder?

kalradivyanshu commented 10 months ago

It will have to be opt in at time of configuring multiple encoders, SVT-AV1 gives --sframe-dist and --sframe-mode, along with --forced-max-frame-width and --forced-max-frame-height which lets the all encoders know that they are in sframe mode with each other. As far as I can tell we need to have these settings on in the beginning, so it cant be like keyframes where we can choose for each frame whether we can make it a keyframe or not.

Regarding decoders, that sounds good, so I just create 1 decoder, and then switch it to the lower layer on sframe and it should just start decoding the lower layer?

kalradivyanshu commented 10 months ago

Also, @sandersdan am i correct in assuming that in the current AV1 encoder in chrome, sframes are turned off i.e. sframe-dist = 0, so in webcodec AV1 encoder, no sframes will be generated?

sandersdan commented 10 months ago

Regarding decoders, that sounds good, so I just create 1 decoder, and then switch it to the lower layer on sframe and it should just start decoding the lower layer?

I believe so; I'll be interested to hear your results.

in the current AV1 encoder in chrome, sframes are turned off

That is correct, the sframe libaom configuration parameters are not currently set by Chrome.

It's worth noting that WebCodecs only really has temporal scalability so far, and this appears to be a knob relating to spatial scalability. We may want to work on the basics first.

kalradivyanshu commented 10 months ago

Thankyou for your reply!

I believe so; I'll be interested to hear your results.

Will get back to you.

It's worth noting that WebCodecs only really has temporal scalability so far, and this appears to be a knob relating to spatial scalability. We may want to work on the basics first.

But adding spatial scalability means adding a lot of extra functionality https://github.com/w3c/webcodecs/issues/619. Switch frames can be added independently, and can work even in the current setup, by just adding a new {av1: {frame_ty: "sframe"}} to the EncodedVideoChunk. This will allow much better implementation of simulcast, and could help fill the gap that exists right now in webcodecs SVC, till full SVC support comes in.

this appears to be a knob relating to spatial scalability

I don't 100% agree? It gives you ability to switch between multiple encoded streams, rather than having one encoded stream. SVT-AV1 doesn't support SVC at all, but supports switch frames. I think switch frame is just better simulcast than being a knob in spatial SVC (i.e. its a simulcast feature, not an SVC one, if that makes sense). I might be wrong though, I am very new at all of this.