WebCodecs h264 hardware decode buffers output until next keyframe introducing unnecessary latency

[redonat]

h264 hardware decoder buffers the frame output until the next keyframe is fed(with decode), i.e.:

For a fixed GOP-size of 4, the timeline of events is as follows:

call decode(key0)   in frame0
call decode(delta1) in frame1
call decode(delta2) in frame2
call decode(delta3) in frame3

call decode(key4)   in frame4
output(key0)   called in frame4
output(delta1) called in frame4
output(delta2) called in frame4
output(delta3) called in frame4

call decode(delta5) in frame5
call decode(delta6) in frame6
call decode(delta7) in frame7

call decode(key8)     in frame8
output(key4)   called in frame8
output(delta5) called in frame8
output(delta6) called in frame8
output(delta7) called in frame8

This behavior is consistent with other fixed and variable-rate GOP-sizes we've tested both on NVIDIA and Qualcomm(Android). The decoder should be able to decode the keyframe independently and we've confirmed that the stream is free of B-frames (i.e. no individual frame depends on the next keyframe).

This behavior doesn't exist on the software decoder, it outputs the decoded frames ASAP without waiting for the next keyframe.

optimizeForLatency is set to true - yet the hardware decoder is prioritizing throughput over latency which seems like a bug.

We have ultra-tight latency constraints so this is a big deal for us, any ideas why this might be happening on hardware decode but not on software?

[redonat]

Also filed in https://bugs.chromium.org/p/chromium/issues/detail?id=1352442, was unsure where to submit this.

[sandersdan]

WebCodecs doesn't provide latency guarantees. Chrome however does intend to provide reasonable performance, will continue discussion there.