Opus Packet Loss Concealment

[rvadhavk]

From the libopus documentation, it looks like the way to handle missing packets and have the decoder output packet loss concealment is to pass it an empty/null buffer.

But when creating an EncodedAudioChunk with a zero-length buffer and passing it to the decoder, in Chrome I get an error of "DOMException: Null or empty decoder buffer." The check in Chrome is here.

Is this the expected behavior, a Chrome implementation issue, or is this behavior something that should be clarified in the standard?

[bdrtc]

the same issue exist for decode opus inband fec when packet loss, chrome does not impl this and no describe for such scenario in standard。

[dalecurtis]

@sandersdan @tguilbert-google

[sandersdan]

[Edit: replacing comment]

This doesn't appear in the spec, not sure why Chrome is enforcing it.

I filed https://bugs.chromium.org/p/chromium/issues/detail?id=1363687

[sandersdan]

Reopening as a discussion topic: should WebCodecs be exposing loss concealment, and if so is an empty buffer the right API?

[dalecurtis]

If this is how the opus library has standardized signaling and it's what developers / RTC implementations are most familiar with, we probably should. I expect the null packet does something better than what would happen if we required developers to call flush() and skip these packets?

[aboba]

Agree with @dalecurtis. The null packet will indeed do something better than calling flush(), whether that is concealment or potentially enabling recovery when FEC is enabled.

Note: Opus FEC is distance 1 so it cannot handle burst loss. As a result redundant audio (RED) may also be desirable, but that is typically handled in the application, and therefore probably isn't needed in WebCodecs. For a discussion of RED, see: https://webrtchacks.com/red-improving-audio-quality-with-redundancy/

[sandersdan]

I've been testing in Chrome and it does not appear to be possible to activate this feature in libopus via ffmpeg, as ffmpeg interprets an empty packet as EOS. This likely means that the feature will not be implemented in Chrome in the short term.

[dalecurtis]

Editor's call: Allow null packets for opus. Editors were not aware of any other codecs which have this mechanism, so not worth a new encoded audio chunk type or similar. Probably worth a note in the registry that this is how FEC works for opus.

[sandersdan]

FYI: There is a proposed FFmpeg patch (appears to be abandoned) to implement this API; that patch worked by turning PTS discontinuities into FEC / error concealment.

[aboba]

@dalecurtis Should this be labeled "Extension"?

[padenot]

Related to #827, but not duplicate.

[aboba]

The Opus 1.5 new features specified by IETF MLCODEC WG (such as Deep PLC and Deep RED) depend on being able to indicate lost frames to trigger concealment and recovery. If this is not supported by providing a null/empty buffer, can it be indicated through another mechanism such as a gap in the timestamp?