search

Placeholder-Software / Dissonance

Unity Voice Chat Asset
71 stars 5 forks source link

[bug] Malfunctioning AEC and Buffer Overflow in Dissonance Demo for Mirror Networking #178

Closed konradcl closed 4 years ago

konradcl commented 4 years ago

Context

I was following the guide in the Dissonance documentation for how to setup Acoustic Echo Cancellation. While trying to verify that the Acoustic Echo Cancellation works as intended, I attempted to select the "Dissonance Echo Cancellation" filter that I previously had attached to an audio mixer. This attempt failed and resulted in a NullReferenceException and an Assert failed error repeatedly being logged to the Unity console. If I at this point would exit and reenter play mode, the Unity editor would crash to desktop.

The details of how to reproduce the above problem are outlined in this GitHub issue. In the comments of that issue, a work around has been suggested. Namely, one can remove the if-statement containing a call to GetFloatBuffer in the DissonanceAecFilterInspector class. This work around fixed the immediate issue described above, but I am still encountering problems down the road, which might be related to the NullReferenceException bug. Specifically, Dissonance's Acoustic Echo Cancellation (AEC) fails to prevent echo from in-game music (even of low, but clearly audible, volume), distorts microphone audio, and gives rise to buffer overflow warnings.

I have reproduced the above behavior in Dissonance Demo for Mirror Networking and explained how in the section "Steps to Reproduce".

Expected Behavior

I expect Dissonance AEC to predominantly prevent all echo caused by in-game music being fed back through a microphone. Moreover, I expect that the AEC should prevent echo caused by speaking into the microphone in a conversational manner.

Actual Behavior

After following the steps outlined in "Steps to Reproduce", letting in-game music be fed back via the microphone gives rise to painfully loud echo. Furthermore, should you speak into the the microphone, you will hear that the played back speech is distorted and barely audible; it may also give rise to echo. Finally, if the demo client (built app) and server (Unity editor) is left running for a few minutes, buffer overflow warnings, as the subsequently shown, will usually be logged to the Unity console.

Buffer Overflow Warning

[Dissonance:Recording] (19:04:06.075) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate
UnityEngine.Debug:LogWarning(Object)
Dissonance.LogMessage:Log() (at Assets/Libraries/Plugins/Dissonance/Core/Log.cs:63)
Dissonance.Logs:SendLogMessage(String, LogLevel) (at Assets/Libraries/Plugins/Dissonance/Core/Log.cs:95)
Dissonance.Log:WriteLog(LogLevel, String) (at Assets/Libraries/Plugins/Dissonance/Core/Log.cs:178)
Dissonance.Log:WriteLogFormat(LogLevel, String, Int32) (at Assets/Libraries/Plugins/Dissonance/Core/Log.cs:187)
Dissonance.Log:Warn(String, Int32) (at Assets/Libraries/Plugins/Dissonance/Core/Log.cs:390)
Dissonance.Audio.Capture.BasePreprocessingPipeline:Dissonance.Audio.Capture.IMicrophoneSubscriber.ReceiveMicrophoneData(ArraySegment`1, WaveFormat) (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/BasePreprocessingPipeline.cs:186)
Dissonance.Audio.Capture.BasicMicrophoneCapture:SendFrame() (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs:378)
Dissonance.Audio.Capture.BasicMicrophoneCapture:ConsumeSamples(ArraySegment`1) (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs:336)
Dissonance.Audio.Capture.BasicMicrophoneCapture:DrainMicSamples() (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs:308)
Dissonance.Audio.Capture.BasicMicrophoneCapture:UpdateSubscribers() (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs:243)
Dissonance.Audio.Capture.CapturePipelineManager:Update(Boolean, Single) (at Assets/Libraries/Plugins/Dissonance/Core/Audio/Capture/CapturePipelineManager.cs:187)
Dissonance.DissonanceComms:Update() (at Assets/Libraries/Plugins/Dissonance/DissonanceComms.cs:608)

While I on a regular basis encounter said warnings, I cannot pinpoint how to invoke them. Often, letting the client and server run for a few minutes gives rise to these warnings, but not always. Sometimes, the warnings are triggered by clicking on a game object in the hierarchy pane. (I think the game object needs to, in some way, be connected to the Audio Echo Cancellation filter, but I have not tested this throughly.) Finally, I have repeatedly generated the warnings by moving a program window, such as that of a text editor, into the desktop in which the Unity editor is located.

Steps to Reproduce

Before following the below steps, follow the steps to reproduce the NullReferenceException bug mentioned in the "Context" section. These steps can be found by visiting the bug's GitHub issue and navigating to its "Steps to Reproduce" section. Alternatively, you can download a demo project with in which the NullReferenceException bug is already reproduced by clicking here. At the very end of this issue there also exists a link to download a demo project in which the below steps have been completed. This project can be used to immediately check if the behavior on your machine matches the behavior described in the "Actual Behavior" section.

To reproduce the bug, complete the below steps. Always save the changes that you perform in the Unity editor. Use a set of speakers (not headphones) and do not change your audio output or input device while trying to reproduce this bug.

  1. Comment out the if-statement calling GetFloatBuffer in DissonanceAecFileInspector.cs (lines 56-88)
  2. Download this music file by clicking here.
  3. Add the music.wav file to the Unity Assets folder.
  4. Add the "MirrorIgnorace Game World" scene to Unity's hierarchy pane.
  5. Add an AudioSource component to the "MirrorIgnorance GameWorld" scene.
  6. Attach the music file imported in step three to the AudioClip property of the AudioSource added in the previous step.
  7. Check the AudioSource's "Loop" property.
  8. Assign the mixer containing the "Dissonance Echo Cancellation" filter to the Output property of the AudioSource added in step four.
  9. Add the "MirrorIgnorance Demo" scene to Unity's hierarchy pane and remove the "MirrorIgnorance Game World" from the same pane.
  10. Build and run the Unity project to Mac OS X.
  11. Authorize the built app to do whatever it asks (if anything).
  12. Enter play mode in the Unity editor.
  13. When in play mode, press the "LAN Host" button in the game window.
  14. Mute the audio from Unity by pressing the "Mute Audio" button above the game window.
  15. Press the "LAN Client" button in the app built in step ten.
  16. Ensure that the volume on your computer is turned on, and let the in-game music play for roughly 10 seconds. You can also try speaking into your microphone.

Following these steps should result in the behavior described in the "Actual Behavior" section.

Environment

The described issue was initially encountered and later reproduced in the following environment:

martindevans commented 4 years ago

This was also tested in the built player (discussed on Discord), same issue.

martindevans commented 4 years ago

Tom reproduced this issue but after some discussion we think it may be expected behaviour: the AEC is not expected to function correctly if the microphone of one client can record the speakers of another client (this thoroughly confuses the AEC). In this test case the clients can obviously hear one another - could this also be the problem when testing the AEC in your real app?

martindevans commented 4 years ago

I'll close this issue, since we think it wasn't a bug in this case. Feel free to re-open it if it's still a problem :)