Implement simplest "beep" and "echo" PLC

[gavv]

Intro

Packet loss concealment (PLC) is a technique to mask the effects of packet loss. It comes into play when a packet is lost and can't be recovered using FEC. In this case we can at least make the loss less noticeable.

Currently we just play zeros instead of the lost samples. The simplest PLC technique is just to repeat last frame, but a little quieter. In this task we want to implement this technique.

We also have "beeping mode", useful for debugging. In this mode, when a packet is lost, we insert a loud beep instead of zeros, to make the loss more noticeable, which also may be considered as a kind of PLC. This feature is currently implemented as a part of Depacketizer class. In this task we want to extract it to a separate PLC-like element.

Task

• First we should finish #614 and #615 because we need partial reads support to implement PLC. See below why.

• Then we should modify audio::Depacketizer.

  Depacketizer makes audio frames from network packets. It decodes samples from packets and writes them into frames. It also sets different frame flags.

  The two important flags here are FlagBlank and FlagIncomplete. FlagBlank is set when the frame is completely filled with zeros because of packet loss. FlagIncomplete is set when frame is partially filled with zeros because of packet loss.

  On a regular loss ratio, most frames are fully complete, some frames have FlagIncomplete, and almost no frames have FlagBlank. But the problem with FlagIncomplete is that we can't say what samples in frame are zero because of packet loss, and what samples are zero because they are really zero. But we need to distinguish these cases in PLC.

  To solve this, we should employ partial reads in Depacketizer. Depacketizer should ensure that the frame will be either fully complete or fully blank. FlagIncomplete should be removed at all.

  Here is how it will work. Assume that the stream is the following: 30 samples received, 40 samples lost, 30 samples received. The user requests Depacketizer 100 samples. Depacketizer will return only 30 samples (without FlagBlank). The user then requests 70 samples (100-30). Depacketizer will return only 40 samples (with FlagBlank). Finally, the user requests 30 samples (100-30-40). Depacketizer returns 30 samples (without FlagBlank).

  With this approach, each returned is fully described with its flags. Each frame is either complete, and there is no need for PLC, or blank, and should be recreated using PLC.

  We should also cover this feature in Depacketizer unit tests.

• Then we should add a new pipeline element, audio::EchoPlcReader. Its read() operation will work as follows:

  • Read frame from nested reader.

  • If the frame doesn't have FlagBlank flag, return the frame as is. But before returning, also copy samples from the frame to an internal fixed-sized ring buffer.

  • If the frame has FlagBlank and has size N, fill the frame with the last N samples from the internal ring buffer, with a bit lowered volume. If N is larger than the ring buffer size, the ring buffer contents can be repeated.

  We should also add unit tests for EchoPlcReader.

• Similarly to EchoPlcReader, add BeepPlcReader. Its implementation is ever simpler. If the frame is blank, just fill it with a loud beep.

  Beeping code should be extracted from Depacketizer. And Depacketizer's beep option should be removed.

  We should also add unit tests for BeepPlcReader.

• Integrate EchoPlcReader and BeepPlcReader into the pipeline.

  Add it to pipeline::ReceiverSession. Enable it conditionally, only if enabled in pipeline::ReceiverSessionConfig. By default, EchoPlcReader should be enabled.

  Remove "beeping" boolean from ReceiverSessionConfig. Instead, add "plc_engine" field with three possible values: PLC_Echo, PLC_Beep, PLC_None.

• Add a PLC test to roc_pipeline unit tests.

• Add command-line option to choose PLC mode. Three modes should be present: "none", "echo", and "beep". "echo" should be default.

• Add PLC support to public API. Add roc_plc_engine enum with values ROC_PLC_DISABLE (-1), ROC_PLC_DEFAULT (0), and ROC_PLC_ECHO (1), and ROC_PLC_BEEP (2), similar to roc_resampler_profile. Add plc_engine field to roc_receiver_config. Pass it from API to pipeline config.

Notes

It probably makes sense to split the implementation into a few pull requests:

• implement partial reads in Depacketizer;
• add EchoPlcReader and BeepPlcReader;
• integrate PLC into pipeline, command-line and API.

Info

Related docs: https://roc-streaming.org/toolkit/docs/internals/data_flow.html

[fusuiyi123]

I can take a look at this:D

[gavv]

You're welcome.

[fusuiyi123]

Hi @gavv if I understand correctly, https://github.com/roc-project/roc/blob/develop/src/modules/roc_audio/depacketizer.cpp#L108-L114 each time we should either read_missing_samples_ or read_packet_samples_ so that each time it is either FlagBlank or not FlagBlank?

[gavv]

each time we should either read_missingsamples or read_packetsamples

Yes. Note that during each Depacketizer::read() we may call read_packetsamples() and read_missingsamples() multiple times, depending on whether we have gaps (missing packets) and how much.

so that each time it is either FlagBlank or not FlagBlank?

Not quite.

We set FlagBlank if the frame was completely filled with read_missingsamples(). We set FlagIncomplete is the frame was partially filled with read_missingsamples() and partially with read_packetsamples(). And we set none of these two flags if the frame was completely filled with read_packetsamples().

BTW, take a look at depacketizer unit tests, they may help to understand its behavior.

[gavv]

BTW, forgot to mention, did you see this notice in the issue?

First we should finish #302 because we need partial reads support to implement PLC. See below why.

302 is in progress, @aj-thomas-8 works on it.

[fusuiyi123]

yes I saw #302 , cool I will take a look at the unit test:)

[fusuiyi123]

since this is a task after #302 may I look at this other issues first:)

[gavv]

302 is unassigned again

[gavv]

Moved to #731