search

roc-streaming / roc-toolkit

Real-time audio streaming over the network.
https://roc-streaming.org
Mozilla Public License 2.0
1.08k stars 213 forks source link

Add Opus support #51

Open gavv opened 8 years ago

gavv commented 8 years ago

Opus is a lossy streaming audio codec.

Specs:

Adding Opus support introduces a few challenges:

Our fec::Decoder is packet-format-independent and works before decoding audio packet payload. Thus, it theoretically may be used in conjunction with Opus decoder, if it makes sense.

baranovmv commented 8 years ago

Opus decoder is stateful

How may it impact on code? And what does it mean literally? Do we have to provide every packet consecutively? We do so already. Are we disallowed to loose packets? This would mean that opus is designed for unrealistic channels.

Opus decoder combines them

I thought this is fec::Decoder's destiny.

gavv commented 8 years ago

How may it impact on code? And what does it mean literally?

Currently Streamer extracts samples from every packet independently using packet->read_samples().

With Opus, we need a different approach: Streamer (or anybody else who needs extracting samples) should ask opus decoder to extract samples from next (possibly lost) packet.

Are we disallowed to loose packets? This would mean that opus is designed for unrealistic channels.

No, of course.

I thought this is fec::Decoder's destiny.

It seems that Opus API combines decoding samples and repairing lost packets. Thus, if we need repaired packets before decoding samples (i.e. before Streamer), we probably should decode samples before Streamer and store them somewhere.

I don't think we could (and need) reuse fec::Decoder for Opus support. They're working on different layers: fec::Decoder works on packet layer and is payload-independent, while opus decoder actually works on audio stream layer.

gavv commented 8 years ago

One more issue (maybe):

opus_decoder_init()

Sampling rate to decode to (Hz). This must be one of 8000, 12000, 16000, 24000, or 48000.

baranovmv commented 8 years ago

May we make a special handle for Opus that looks like Reader of audio packets?

If Opus do FEC, we don't have to invoke OpenFEC and others. If not, we can and have to use fec::Decoder and Opus. I think we have to provide arguments letting user determine usage of third party stand alone FEC and Opuses FEC.

gavv commented 8 years ago

If Opus do FEC, we don't have to invoke OpenFEC and others. If not, we can and have to use fec::Decoder and Opus. I think we have to provide arguments letting user determine usage of third party stand alone FEC and Opuses FEC.

Agree. BTW, it may be interesting to combine them together.

May we make a special handle for Opus that looks like Reader of audio packets?

Not sure. In this case:

In other words, we pass packets to Opus decoder and it produces (virtually) audio stream. Then we split this stream on temporary (fake) packets and pass them to Streamer. And finally Streamer concatenates those packets into audio stream again. It makes little sense.

But we can avoid this:

  1. We introduce a concept of (stateful) AudioSampleDecoder (allowed to perform FEC) which should be used in Streamer. This concept may be useful in future if we'll add support for more stream-level codecs.
  2. Alternatively, we can implement OpusStreamer that replaces our regular Streamer when we're using Opus.
gavv commented 8 years ago

Do we have any components with hard-coded sample rate of 44100Hz?

baranovmv commented 8 years ago

We have no places to mention sample rate anywhere except alsa driver or this kind of things

gavv commented 8 years ago

Great.

gavv commented 8 years ago

Another important note: we create Streamer for every channel and deinterleave packets before Streamer, while Opus decoder is used for all channels and we should deinterleave samples after it.

gavv commented 5 years ago

Note on FEC in Opus: http://lists.xiph.org/pipermail/opus/2013-January/001904.html