Consider Opus FEC and PLC

[ikonst]

A good way to experiment is:

gst-launch filesrc location=foo.wav ! wavparse ! audioresample ! \
  opusenc inband-fec=1 packet-loss-percentage=10 ! \
  identity drop-probability=0.1 ! \
  opusdec use-inband-fec=1 plc=1 ! \
  autoaudiosink

plc=1 definitely improves things, while package-loss-percentage > 0 just seems to make things worse, so this should be researched.

Also, see https://bugzilla.gnome.org/show_bug.cgi?id=744338. Before opusdec gains a way of knowing the actual maximum packet duration, the constant 120ms latency added by use-inband-fec=1 would make it inappropriate for RTC.

[ikonst]

@sdroege , you might have some input.

[superdump]

I'm not sure about signaling for in-band opus FEC. But fine for non-interoperable usage.

This draft is being discussed by IETF at the moment. https://tools.ietf.org/html/draft-ietf-payload-flexible-fec-scheme-00

[ikonst]

I'm not sure about signaling for in-band opus FEC.

FEC's presence is signaled by Opus headers, so it's naturally interoperable; i.e. if you start sending Opus in-band FEC while your decoder doesn't have use-inband-fec=0, nothing will break, and vice versa.

plc=1 is also not an interop risk.

The maximum packet duration corresponds to SDP's maxptime according to Oliver Crête, so this is already standardized.

[ikonst]

Anyway, I'd appreciate if somebody else will try, e.g.:

gst-launch-1.0 filesrc location=/Users/ilyak/music.wav ! wavparse ! audioresample ! \
  opusenc inband-fec=1 bitrate-type=0 bitrate=64000 packet-loss-percentage=25 ! \
  identity drop-probability=0.2 ! \
  opusdec use-inband-fec=1 plc=1 ! \
  autoaudiosink

and confirm or disprove my impression that Opus FEC doesn't really work.

• Without drop-probability > 0, setting inband-fec=1 results in slightly lower quality, which is expected since some of the bitrate is being wasted on FEC. This brings me to assume FEC is indeed being added.
• With drop-probability > 0, I've tried setting packet-loss-percentage slightly higher than the drop probability (e.g. 20 --> 25). I'd assume coding theory stuff would make sure to add just enough FEC to counter the drops, but what I'm hearing is not clearly better. In fact, plc=1 seems to be what's making a difference. With plc=0, it's really bad.

Ideas, anyone?

[ikonst]

OK, actually read the source.

• plc=1 actually affects GstAudioDecoder and makes it send an empty buffer for GAP events (which identity drop-probability=... generates). For Opus to trigger PLC, it simply needs a NULL buffer, which plc=1 results in eventually.
• Without those gap (= zero-length) buffers, FEC reconstruction would not be triggered either. Therefore use-inband-fec=1 plc=0 is meaningless.
• Furthermore, FEC reconstruction seems broken — see https://bugzilla.gnome.org/show_bug.cgi?id=753148