Asio & Opus

[CyrilleFormont]

Hi Mark, First of all, thank you for your great work!

We created an application that allows users to communicate through an UDP Server using NAudio. We started by creating a P.O.C using WaveIn and WaveOut, we encoded the byte[] using Opus, send that through to the server, the clients received it, that worked. However, the latency was a bit too high for us using WaveIn/Out so we decided to use Asio instead.

We then changed our way to do, we are now using Asio, we convert the float[] to an byte[], send that through the network, the clients gets it, that works perfectly, the latency is perfect and the sound is crystal clear. Only issue there, is that the bandwidth is obviously busy by this transfer of data as we aren't encoding the byte[].

We then tried to encode it using Opus, send it, decode it then, the sound becomes super distorted.

Is there something that comes-up to your mind ?

Here some part of the code that might help:

Sender, Init

            var buffered = new BufferedWaveProvider(IeeeFloatWaveFormat);
            _asioOut = new AsioOut(driverSelected);
            _asioOut.InitRecordAndPlayback(buffered, 1, codec.RecordFormat.SampleRate);
            _asioOut.AudioAvailable += _asioOut_AudioAvailable;
            _asioOut.Play();

Sender, Asio Callback method

        private void _asioOut_AudioAvailable(object sender, AsioAudioAvailableEventArgs e)
        {
            var samples = new float[e.SamplesPerBuffer * e.InputBuffers.Length];
            e.GetAsInterleavedSamples(samples);
            var byteArray = new byte[samples.Length * 4];
            Buffer.BlockCopy(samples, 0, byteArray, 0, byteArray.Length);

            var soundBuffer = new byte[byteArray.Length + _notEncodedBuffer.Length];
            for (var i = 0; i < _notEncodedBuffer.Length; i++)
                soundBuffer[i] = _notEncodedBuffer[i];
            for (var i = 0; i < byteArray.Length; i++)
                soundBuffer[i + _notEncodedBuffer.Length] = byteArray[i];

            var byteCap = _codec.FrameByteCount(_codec.SegmentFrame);
            var segmentCount = (int)Math.Floor((decimal)soundBuffer.Length / byteCap);
            var segmentsEnd = segmentCount * byteCap;
            var notEncodedCount = soundBuffer.Length - segmentsEnd;
            _notEncodedBuffer = new byte[notEncodedCount];
            for (var i = 0; i < notEncodedCount; i++)
                _notEncodedBuffer[i] = soundBuffer[segmentsEnd + i];

            for (var i = 0; i < segmentCount; i++)
            {
                var segment = new byte[byteCap];
                for (var j = 0; j < segment.Length; j++)
                    segment[j] = soundBuffer[(i * byteCap) + j];
                var buff = _codec.Encode(segment, segment.Length, out var len);
                var list = buff.ToList();

                var bytes = BitConverter.GetBytes(this.UserId).ToList();
                list.RemoveRange(len, (buff.Length - len));
                bytes.AddRange(list);
                _audioSender.Send(bytes.ToArray());
            }
        }

Player, Init

           _waveProvider = new MixingSampleProvider(IeeeFloatWaveFormat);
            _codec = codec;
            _receiver = receiver;
            TextReceived += callback;
            receiver.OnReceived(OnDataReceived);
            var stereo = new PanningSampleProvider(_waveProvider)
            {
                Pan = 1f
            };
            _output = new AsioOut(driverSelected);
            _output.Init(stereo);
            var playing = new Thread(_output.Play);

            this.AddClient(0);

            playing.Start();

Player, OnDataReceived method

        private void OnDataReceived(byte[] compressed)
        {
            try
            {
                var decoded = _codec.Decode(compressed, compressed.Length,out var length);
                var client = this.Clients.FirstOrDefault(c => c.UserId == 0);
                client?.WaveProvider.AddSamples(decoded, 0, length);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                var data = Encoding.ASCII.GetString(compressed);
                TextReceived?.Invoke(data, null);
            }
        }

WaveFormatCreated

WaveFormat IeeeFloatWaveFormat => WaveFormat.CreateIeeeFloatWaveFormat(this.RecordFormat.SampleRate, 1);

[CyrilleFormont]

Anyone has an idea on that ?

[markheath]

what format does the opus decoder decode into? I'd guess 16 bit PCM. It should match the format of the BufferedWaveProvider you are writing to or it could be the encode stage. You need to supply the opus encoder with exactly the input format that it is expecting

I recommend doing the following:

1. using ASIO save audio to a WAV file. Check the WAV file sounds OK
2. write code to convert the WAV file into opus, and then convert back to WAV. Check that sounds OK
3. once you are sure your encode decode works correctly, then plug it back into the network + ASIO code

[CyrilleFormont]

That sounds like a plan @markheath, thank you!