[BUG] high latency for inbound audio track

[AndrewMeadows]

We're seeing high latency (up to 2.0 seconds) between speaking into the microphone of the remote peer and playing the audio at the receiving peer . We think it is caused by a growing webrtc jitter buffer for the inbound audio track, however we are still trying to investigate and learn more.

To Reproduce (1) Unity Peer1 on machine1 connect to C++ webrtc server (2) Chrome browswer Peer2 on machine2 connect to C++ webrtc server (3) Note: server does spatialization mixing and relays results to Peer1 and Peer2 (4) Peer1 speaks into mic... (5) Peer2 hears audio with low latency (0.25 - 0.5 seconds) (6) Peer2 speaks into mic ... (7) ... Peer1 hears audio with high latency (1.0 - 2.0 seconds) BUG!

It is intermittent! Sometimes we can get a lower latency (0.25 - 0.5 seconds) by doing the following: (8) Mute the mic on peer2 and wait several seconds. (9) Unmute the mic on Peer2 and speak... (10) ... Peer1 sometimes hears audio with lower latency but eventually the latency grows to high levels again.

Note: we also tested on Windows but there only noticed high latency: we couldn't use the mute/unmute trick to get intermittent low latency audio. We didn't try this for very hard: two or three times. We mostly just wanted to verify that it wasn't a linux-only problem.

Expected behavior The latency for audio from Peer1 to Peer2 is low (0.25 - 0.5 seconds). We would expect similar latency in the other direction: Peer2 to Peer1.

Theory: is Unity draining webrtc's audio buffer too aggresively? The behavior suggests the unity plugin is sometimes asking for audio from webrtc's inbound audio buffer too often, or in a bursty fashion, which could cause webrtc to supply "fake" audio data and grow its jitter buffer to compensate. When inbound audio is muted webrtc has an opportunity to flush its jitter buffer to be smaller which is maybe why it sometimes shows up as low latency for a while.

We are trying to investigate. The plan is to harvest webrtc stats to make a histogram of jitter buffer state. If we learn more we will post updates.

Environment:

• OS: Ubunutu-20.04
• Browser: Chrome 92.0.4515.159 (for Peer2)
• Unity Version: 2020.3.17f1
• Package version: 2.4.0-exp.4

[karasusan]

We recognize the issue that the audio streaming latency. It is so helpful and waiting for your additional information.

[tomires]

Hello, I have also encountered this issue. In addition to latency, parts of the incoming stream get repeated every ~5s making the stream barely usable (outgoing stream works just fine). We have tested this against our WebRTC implementation running on a webserver, but the same issue is present in the "Audio" example. Testing was done on iPad Pro 4th gen - please see the following example (I have slightly modified InputAudioStream by changing the type to 3D and placing it away from the camera so that we don't hear everything twice)

Archive.zip

I assume it is the same issue, but if you want I can open a separate bug report.

Environment: OS: iOS 14.2 Unity Version: 2020.3.0f1 Package version: 2.4.0-exp.4

[AndrewMeadows]

TL;DR AFAICT the latency is not caused by a large WebRTC jitter buffer. It is something else.

I instrumented my client to grab WebRTC stats. Using the documentation about the various stats I was able to compute the latency caused by the jitter buffer itself. I plotted the latency and some other stats in real-time to see how they correlate with the latency I'm experiencing. Here is a typical graph at startup: The graph is showing four distinct values together. The plots are scaled to span the full vertical range of the graph so the maximum value of each data set is printed at the top in corresponding color. The latency (white) plot has units of "seconds" and its peak in this snapshot was almost 50msec. It would remain around 50msec even when I heard latency of a second or more. I could not find evidence that the WebRTC jitter buffer is being terribly mishandled by Unity.

However, I could make the jitter buffer grow by moving the Unity Editor window out of focus. I'm using the Fxce Window Manager on Linux and when I shift mouse/keyboard focus to another window it will deschedule the Unity Editor window, making its frames slow down. And when I do that the WebRTC jitter buffer will grow. In the plot below I moved focus away from and back to the Unity Editor a few times. However, even though the jitter latency grows to about half a second it doesn't really change the qualitative latency I'm hearing. It is a little difficult to measure latency accurately with an untrained ear, and I'm not well calibrated, but I don't think I would be able to tell with my eyes shut whether the Unity Editor window had focus or not. Can I tell the difference between 1.0 and 1.5 seconds of latency? Probably not.

If anyone is interested in reproducing these results: the code is open and I could supply instructions on how to run it. It would require someone to make a free demo account on the HighFidelity.com and setup a test space and test Javascript Web Token (JWT).

[tomires]

Made an example video to illustrate the issue. Hopefully it is the same issue as Andrew has discussed. In addition to latency, I am getting occasional repeats and echo.

Reference: https://www.youtube.com/watch?v=NI12WvjBFG4 Output: https://user-images.githubusercontent.com/7209990/134001222-3b8182ff-8427-4948-82e3-2258c8e77607.mp4

Tested on a built-in microphone of 2017 MBP, but I am getting the same issue on Windows (Editor+runtime) and iOS.

[AndrewMeadows]

@tomires, I watched the video. It looks to me like you have an open mic problem but with very high latency. If you isolate your mic from your speakers (i.e. use headphones) I would expect the echo to go away.

[howard-stearns]

If Andrew's idea is correct, then that would mean that @AndrewMeadows and @tomires examples are likely both illustrations of one underlying problem (extremely long latency) rather than there being two different problems (latency and echo/repeats).

[tomires]

I have been using headphones when recording my example. The echo is only a minor problem - the main issue being repeats and subsequent skips, please check the following timestamps in my video:

0:12-0:15 - repeat #1 0:20-0:22 - repeat #2 0:26-0:28 - repeat #3 0:30-0:32 - skip

[karasusan]

@leejeonghun requested to review from us the fix here. If some people are interested in this fix, please review it. thanks. https://github.com/Unity-Technologies/com.unity.webrtc/pull/541

[IdolMenendez]

I mentioned this bug from #538 。Here is some details. I Build Two Server -> signal(c#) and stun Two Client -> pushclient and pullclient

I add two track's at one peerconnection on both client.I did received the video and audio stream and two client could do some communicate. But the Problem is : the audio always later than the video. So the feeling is becoming weird.Usually I say something ,my voice always quicker than my voice.And it's easy to notice. So I think ,may be you can solve this bug.and it's better to give an example of audio-video scene. I think that'll be very helpful. Thank you so much for create this repo. Here is some my client code.may be it can offer some infomation.

    _peerConnection = new RTCPeerConnection(ref config)
    {
        OnTrack = evnt =>
        {
            if (evnt.Track is AudioStreamTrack audio)
            {
                //otherAudioSource = audio.Source;
                audio.OnAudioReceived += (renderer) =>
                {
                    outputAudioSource.clip = renderer;
                    outputAudioSource.loop = true;
                    outputAudioSource.Play();
                };
                Debug.LogError("Poll:otherAudioSource");

            Debugger.LogIfLevel(LoggerType.Debug, () => $"Add Track:{evnt.Track}");
            if (!(evnt.Track is VideoStreamTrack video) ) return ; 
            if( video.IsDecoderInitialized ) return;
            otherView.texture = video.InitializeReceiver(1280, 720);

            evnt.Streams.First().OnRemoveTrack = ev =>
            {
                Debugger.LogIfLevel(LoggerType.Debug, () => $"Un track");
                otherView.texture = null;
                ev.Track.Dispose();
            };
            Debugger.LogIfLevel(LoggerType.Debug, () => $" video :{video.Id}");
        },
        OnIceCandidate = ice =>
        {
            _channel.Route(remoteId, new IceCandidate {IceCandidate_ = JsonUtility.ToJson(ice)});
        },
        OnConnectionStateChange = state =>
        {
            Debugger.LogIfLevel(LoggerType.Debug, () => $"State:{state.ToString()}");
        },
        OnNegotiationNeeded = () =>
        {
            Debugger.LogIfLevel(LoggerType.Debug, () => $"OnNegotiationNeeded()");

        },
        OnDataChannel = (channel) =>
        {
            //_dataChannel = channel;
            channel.OnMessage = (msg) =>
            {
                var any = Any.Parser.ParseFrom(msg);
                Debugger.Log(any);
            };
        }

    };

    var videoTrack = new VideoStreamTrack(webCamTexture);
    _peerConnection.AddTrack(videoTrack);

    _sendStream = new MediaStream();
    m_audioTrack = new AudioStreamTrack(inputAudioSource);
    _peerConnection.AddTrack(m_audioTrack, _sendStream);

    var initData = new RTCDataChannelInit();
    _dataChannel =_peerConnection.CreateDataChannel("Data", initData);

    _channel.OnOffer.Subscribe(d =>
    {
        Debugger.Log($"{d.Data.type} sdp {d.Data.sdp}");
        offerDes =  new RTCSessionDescription
        {
            type =  RTCSdpType.Offer,
            sdp =  d.Data.sdp
        };
        remoteId = d.FromID;
        receive .gameObject.SetActive(true);
        return false;
    });

    _channel.OnIceCandidate.Subscribe(d =>
    {
        _peerConnection.AddIceCandidate(d.Data);
        return false;
    });

[tomires]

@leejeonghun requested to review from us the fix here. If some people are interested in this fix, please review it. thanks. #541

Thanks for a reply! I have tested the changes made in @leejeonghun 's PR, unfortunately the performance seems to be similar to what I have described earlier.

https://user-images.githubusercontent.com/7209990/137645113-723559ac-a54a-4a95-87c8-2c096a7e490a.mp4 (reference audio track is the same as before)

[leejeonghun]

@tomires Thank you for testing. I added an additional commit to improve audio quality. I'd like you to test again. It's required build native plugin to test. please check it.

[leejeonghun]

Here's my test result with the same reference as @tomires. https://user-images.githubusercontent.com/11531985/139172792-7beecb25-c89f-48da-a240-f410b28556e7.mp4

[tomires]

Sounds awesome. I will test it out over the weekend :)

[tomires]

@leejeonghun Unfortunately, I am still getting the same issue as before on macOS and iOS. I noticed you were testing on Windows which might explain the different result. Is it possible that I have built the plugin incorrectly? I have used the instructions in README.md and didn't get any error during the build process.

[tomires]

Found the reason - I forgot to clean my xcode build folder. Now I am getting the following result: https://user-images.githubusercontent.com/7209990/139716615-2d0394f1-3067-44ed-8132-b709de1e476e.MP4

Perhaps this is due to the fact I am using Xcode 12... the docs recommend using an earlier version. Unfortunately I am unable to target the current version of iOS using Xcode 11

[AndrewMeadows]

@leejeonghun and @karasusan : I tested the code from #541 and thought it was fixed... but then I created a fresh project and assembled the test from scratch and I realized it was NOT fixed: I was able to reproduce high (~1.5 to 2.0 seconds) audio latency on the inbound audio at the Unity client. The reason I thought it was fixed was user-error: the webrtc-plugin I was using had been added from local-file... which was a copy of my own hacked version of the plugin where I route straight to the default audio device.

[karasusan]

@tomires, @AndrewMeadows Sure. I will push the plugin built on internal CI.

Edited: I pushed native plugins. Please download them and check it. https://github.com/Unity-Technologies/com.unity.webrtc/commit/f31892d197d14d8c826ce19d70de4affebc7214d

[tomires]

I am still getting a crash on iOS after a couple seconds of run time.

As for the macOS package, when I tried running the demo app in editor, I received the following error:

I have tried matching the method signature to its form in the penultimate commit in PR #541 as the second argument was omitted in that one, however I ended up getting the same error.

[leejeonghun]

@tomires The second screenshot, it seems that the native plugin update is not done correctly. Please check Runtime/Plugins/macOS/webrtc.bundle/Contents/MacOS/webrtc is replaced with f31892d correctly.

Unfortunately I don't have a mac device so it's hard to help. @karasusan, can you test with a mac device?

[karasusan]

@leejeonghun I tried it on mac mini (Apple Silicon), and I found the crash bug.

• Unity editor did not crash when testing the video streaming sample.
• After a few seconds from starting audio streaming, Unity editor crashed.

I will test it again on mac book pro(Intel) tomorrow.

[leejeonghun]

I found a silly bug at audio buffer handling. I pushed an additional commit to fix it.

@tomires, after @karasusan push the native plugin built on internal CI later, please test it again. @AndrewMeadows, Thank you for testing. PR #541 focused to improve overall audio playback quality. It removes such as looping glitches, noises due to incorrect audio timing but not the latency perfectly. It would be only reduced than current. Please note it.

[leejeonghun]

https://github.com/Unity-Technologies/com.unity.webrtc/pull/541#issuecomment-960537307 Crash bug is fixed. thanks!

@tomires, native plugins were updated at https://github.com/Unity-Technologies/com.unity.webrtc/commit/f388675db593299a61cf6e8f78eda1f58fa60a13. Please download them and try again.

[tomires]

https://user-images.githubusercontent.com/7209990/140562188-c2dc1f35-c4bc-4238-8fbc-71a3d4c09f61.MOV

Can confirm I no longer get crashes on iOS. Looping issue still occurs, but at what seems to be a greatly reduced rate (it can be noticed in the middle of the video). I am getting weird compression-like artifacts, but overall the audio is considerably more usable than what we had previously!

[IdolMenendez]

Any plan to fix this Audio Problem? It seems like there is still exist...

[karasusan]

@IdolMenendez We are going to merge this fix in a few days. If you try this and report the result, it is very helpful. https://github.com/Unity-Technologies/com.unity.webrtc/issues/525#issuecomment-961588704

[IdolMenendez]

@IdolMenendez 我们将在几天内合并此修复程序。如果您尝试此操作并报告结果，那将非常有帮助。 第525章（评论）

Ok，That's awesome，I'll try when the fix commited。Thanks for your professional work。

[karasusan]

memo: WRS-154

[karasusan]

Merged #541, but the native plugin is not updated on the develop branch yet. We will update them soon.

[jina-jyl]

When will this fix be released in a preview version? I have trouble installing the package from the head of the repo.

[karasusan]

@leejeonghun, @tomires, @AndrewMeadows We will review this change #565 by @leejeonghun, please check it. Native plugins are not update in #565 branch, so I pushed native plugins into this branch. https://github.com/Unity-Technologies/com.unity.webrtc/tree/feature/low_latency

[karasusan]

@jina-jyl The next release date is next month.

[tomires]

@leejeonghun, @tomires, @AndrewMeadows We will review this change #565 by @leejeonghun, please check it. Native plugins are not update in #565 branch, so I pushed native plugins into this branch. https://github.com/Unity-Technologies/com.unity.webrtc/tree/feature/low_latency

Works well on my end! I couldn't notice any echoes/repeats

[karasusan]

merged #565. This change will be released for next release at the end of the month.

[karasusan]

We have some issues relating audio streaming and fixing them. https://github.com/Unity-Technologies/com.unity.webrtc/issues/602

[karasusan]

We merged some fixes above to develop branch, and will release the new version which contains them next week.

[karasusan]

@AndrewMeadows @tomires @IdolMenendez @jina-jyl @howard-stearns We released the new version 2.4.0-exp.5. Please check it out and if your issue is not solved, please notify me.