Deep voice on macOS

[Hugodzl]

Hi there,

There is a weird issue happening where the user on macOS can speak and hear, but the voices the user hears are deformed they sound very deep. The other users on Windows can hear that user properly with no deformation. Only the users on macOS hear voices with a deep voice deformation.

We tried with various settings in Voice Settings, changing the Frame size, noise cancellation, audio quality, etc. No impact

This issue occurs in both the editor on Play mode and in our Build (IL2CPP and Mono). I am using Unity 2019.4.14, but also tried in 2019.4.5 and 2019.4.13 macOS 10.15 Catalina

No errors in the console logs. At some point the user on macOS gets this warning:

Dissonance:Recording] (16:54:41.234) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate

and at one point this:

Dissonance:Playback] (16:34:27.788) EncodedAudioBuffer: Error: Encoded audio heap is getting very large (50 items)! This is probably a bug in Dissonance, we're sorry!

[martindevans]

It looks like this may be caused by unusually low frame rates. What kind of frame times are you getting?

Lost 960 samples in the preprocessor (buffer full)

This happens because too much audio was delivered to the preprocessor within a single frame (or the preprocessor is running slow due to a heavily loaded CPU starving the processing thread).

Encoded audio heap is getting very large (50 items)!

This happens because playback is falling behind, again probably due to slow frame rates causing audio playback to fall behind.

[Hugodzl]

Hi @martindevans , Thanks for getting back to me. We are running the application surprisingly well at around 50 fps and sometimes even up to 90fps with our main project. We tested it on another mac as well and we're getting the same results where the voice is distorted and the same console logs. Our second mac's specs are: CPU 2.5GHz Dual-Core Intel Core i5 GPU: Intel HD Graphics 4000 1536 MB Memory: 16 Gb 1600 MHz DDR3

The CPU usage according to the activity monitor usually fluctuates between 120% and 160% running our project.

What's weird is that even the most basic Dissonance Photon demo scene which contains almost nothing gets also the same results and in that case the cpu usage is much smaller. We also tested our main project on low end windows laptops and we haven't encountered that issue. So far it only seems to happen on mac computers. We use PUN2 and Dissonance + the Dissonance for Photon package and we build with Development build ticked on.

[martindevans]

If you're getting this same thing in the demo scene it can't be related purely to performance - that's a very lightweight scene!

Could you up the Dissonance logging settings (Window > Dissonance > Diagnostics) to Debug for all categories and send me a log? I'll see if I can spot any other clues.

[niksigouin]

Hi @martindevans,

I'm currently working in collaboration with @Hugodzl on this project. Here is the editor log dump after upping the Dissonance logging settings:

Dissonance macOS deep voice.txt

Here is a pastebin link if ever: https://pastebin.com/WLVKsQhT

[martindevans]

Thanks for the log. Unfortunately it doesn't show anything that might explain the issue.

Can you send me a project that reproduces this issue? I'll borrow a macbook to test it out myself. Thanks.

[niksigouin]

@martindevans Unfortunate that the logs didn't give any clue to what the problem is. As for the project, we tried our own project (which is 12GB in size give or take) as well as the default demo scene that comes with the Dissonance package. Both getting the same result with the deep voice.

Give the Dissonance Photon Demo project a try on the Mac and if it comes through without problem, we will send the demo scene from our machine.

[niksigouin]

@martindevans Any updates regarding this issue?

[martindevans]

I've had a similar report of a very similar issue from another person as well, so that at least seems to confirm it as a real issue. I haven't had time to dig into it yet though - I'm planning to make time tomorrow.

[martindevans]

I don't have a reliable reproduction of this, but I think I have tracked it down to the following code in Assets\Plugins\Dissonance\Core\Audio\Playback\SynchronizerSampleSource.cs:

// If the buffer is too small slighty increase the count of samples read (by 0.1ms) Desync compensation will think it is ahead of
// where it should be and will slow down playback, which will cause the buffer to grow.
if (_pipeline.BufferCount < 1)
    _totalSamplesRead += WaveFormat.SampleRate / 10000;

Could you try commenting out this entire block to see if that helps with your issue? Thanks.

[niksigouin]

Hi @martindevans, we tried commenting that section of code on both machines (Windows/Mac) but the audio playback on the Mac still has that deep voice. It seems to fluctuate between that deep sound and a normal(ish) sound, but it's very unstable.

[martindevans]

Thanks for checking that. Unfortunately I still don't have a reliable reproduction, so I'm working in the dark a bit. Could you find the VoicePlayback component for the speaker (there's one per remote speaker, child object of DissonanceComms gameobject) and send me a recording of the inspector? It's shows all the sync stats, so hopefully I can work out some more from that!

If you want a workaround for this issue while I'm investigating further. Open up Assets\Plugins\Dissonance\Core\Audio\Playback\SynchronizerSampleSource.cs again and change this:

public float PlaybackRate
{
    get; private set;
}

To this:

public float PlaybackRate
{
    get { return 1; }
    private set { }
}

This essentially disables the rate adjustment system (hardcode rate=1) so it won't be doing the fine adjustments to keep perfectly in sync. That may cause other issues with quite long uninterrupted streams of speech, but it'll resync every time someone pauses for a breath.

[niksigouin]

Hi @martindevans, we've tried your workaround but the results stay the same. Here is a wetransfer link to the video you've requested with the VoicePlayback component open. Also a ZIP file containing the same video.

https://we.tl/t-TPbJANHjfH

Dissonance Deep Voice Glitch.zip

[martindevans]

Could you make sure to get the SamplePlaybackComponent in the video as well (it's just cut off at the bottom in this video)? It has some more technical stats on the state of the playback system. I forgot to mention this last time, sorry!

[niksigouin]

Here it is! I can record an other take if need be.

Dissonance_DeepVoice.zip

[vladbasin]

@martindevans do you have an idea why this is happening? Looks like I am experiencing the same issue on MacOS. Trying to troubleshoot this while testing on two MacBook Pro. I assume that the feeling of 'deep voice' is because the audio is stretched in time. This usually leads to lower audio pitch and therefore 'deep voice' feeling.

[martindevans]

do you have an idea why this is happening?

Not yet. I'm on holiday until the new year at the moment and this will be the first thing I investigate when I get back.

Could I get you to send me a log with all categories set to Debug (in Window > Dissonance > Diagnostics) as well?

I assume that the feeling of 'deep voice' is because the audio is stretched in time

That's correct. That means that somewhere in the system the sample rate is incorrect, so the audio is played back at the wrong rate and is slowed down.

Dissonance does have a system for stretching/compressing audio to slow/speed it up and keep things in sync which is what I initially assumed was broken. However this workaround should entirely disable that system and didn't fix the issue (I assume you see the same?). So I'm going to have to check out other bits of the system.

What's odd is that a mismatch in sample rate anywhere else would cause a whole host of other warnings (due to buffers filling up as audio is played too slowly). So it's going to have to be something else.

[vladbasin]

whole host of other warnings (due to buffers filling up as audio is played too slowly)

just providing more information for you to investigate

However this workaround should entirely disable that system and didn't fix the issue (I assume you see the same?)

yep, this workaround does not fix the issue

[martindevans]

Thanks for all the extra info. There appear to be two main issues here:

EventQueue: Large number of packets pending dispatch (NNN).

This indicates that NNN packets arrived over the network all in the space of one frame! With default frame size that's 500+ms of audio arriving all at once. Since frame times are ok (as shown at the end of the message) this could indicate a problem on the sending end (see below) or terrible network problems. This issue is responsible for the other "detected oversized buffer before playback" warning as well.

Lost NNN samples in the preprocessor

This indicates that the preprocessor failed to process NNN samples of audio from the microphone. Usually this is due to an overloaded CPU or very bad frame rates (which causes audio to buffer up faster than the preprocessor can handle it). However we can see from some of the other messages that frame times are ok.

Insufficient buffer space, request XXX, clamped to YYY

This means that the microphone system (the Unity Microphone class) supplied XXX samples all at once and Dissonance only has enough buffer space for YYY. In this case the microphone supplied 90240 samples of audio all at once, that's nearly two full seconds!

This leads me to think that there's some kind of issue with the microphone. Could you try something like this script (but using 48000 instead of 44100) and see if it sounds ok (correct pitch, no huge delay)? Of you have another mic that you can plug into your Mac it would be good to try that with that script and with Dissonance too, just for comparison.

[martindevans]

Just another thought: on this other issue you were having issues with Fraction Poor Delays. As I mentioned over there sometimes that can indicate an issue with the audio hardware causing bad delays - it's possible the same hardware issue is causing the huge buffering problems here too.

If you are using a bluetooth headset perhaps try changing Line 79 of Assets/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs from:

sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);

to:

sampleRate = minFreq;

This will sample audio at a lower rate and place less load on the bluetooth radio.

[niksigouin]

Any updates regarding this issue? I've updated the Dissonance asset to the latest release (7.2.0) and I get the same results.

This leads me to think that there's some kind of issue with the microphone. Could you try something like this script (but using 48000 instead of 44100) and see if it sounds ok (correct pitch, no huge delay)? Of you have another mic that you can plug into your Mac it would be good to try that with that script and with Dissonance too, just for comparison.

Using this script gave me a perfect result with only a minor delay (+/-200ms of delay). And using the PUN2 Demo for Dissonance Voice Chat yielded the same results.

If you are using a Bluetooth headset perhaps try changing Line 79... Using a dedicated mic (Generic webcam microphone) also yielded the same results.

Everything seems to lead to the reception and playback on the mac side of things. Let me know if I can provide any further details to help resolve this issue.

[martindevans]

I haven't managed to pin down the issue yet - unfortunately I haven't been able to reproduce it myself so it's tricky to debug! However, I have been working closely with someone else who has this issue - I've given them a patch to try and if that works around the issue it will hopefully give me enough information to develop a proper fix.

[martindevans]

I've worked with some other users who have had this issue to try to track down it down some more. In 7.2.1 and 7.3.0 we fixed several bugs in the dynamic audio synchronisation system and that seems to have fixed the problem. Can you confirm if these changes fix the issue for you too?

[martindevans]

We've had those fixes out for a while and no one else has reported deep voice issues. Hopefully this issue is resolved! Don't hesitate to comment here if there's still a problem even with the latest version (8.0.0)