[Help] insufficient buffer space warning on android when sending audio to HoloLens

[mirzasohailbaig]

I'm getting insufficient buffer space warning on android when trying to send audio over to HoloLens it used to work a while back but it no longer seems to send any audio over.

My setup is fairly simple I have a button that acts like a push to talk but I don't use the CommActivationMode.PushToTalk instead I use the CommActivationMode.Open so when user presses button I set a bool called activateChannel to true and when he releases that button it turns to false. Below is the only code that was changed for dissonance.

private bool IsUserActivated()
        {
            //Test the actual activation systems
            switch (Mode)
            {
                case CommActivationMode.VoiceActivation:
                    return _isVadSpeaking;

                case CommActivationMode.PushToTalk:
                    return Input.GetAxis(InputName) > 0.5f;

                case CommActivationMode.Open:
                    return activateChannel;

                case CommActivationMode.None:
                    return false;

                default:
                    Log.Error("Unknown Activation Mode '{0}'", Mode);
                    return false;
            }
        }

On HoloLens I use SalsaDissonanceLink (with some minor edits) to have a characters mouth move with audio but that seems to work fine when audio actually comes in.

Other than the listed change I've attached a pdf with images of relevant of configuration.

Expected Behavior

voice should go over when user presses button.

Actual Behavior

voice doesn't go over when user presses button and a lot of insufficient buffer space warnings.

Environment

Dissonance version used: e.g. v6.4.1

Unity version: e.g. 2018.4.6f1 Personal

Editor Operating System and version: e.g. Microsoft Windows 10 Pro (x64)

Build Settings: Android image attached in pdf , HoloLens Image attached in pdf below

Configuration.pdf

[martindevans]

insufficient buffer space warning

This usually means one of two things:

1. The microphone is acting incorrectly, e.g. supplying audio at the wrong rate. This is fairly unlikely.
2. The frame rate of the application is bad enough that Dissonance cannot process realtime audio. What is the framerate on the device, and do you have any other warnings? (Could you post/link a full log with Debug set for all log categories in Window > Dissonance > Diagnostics).

I have a button that acts like a push to talk but I don't use the CommActivationMode.PushToTalk instead I use the CommActivationMode.Open so when user presses button I set a bool called activateChannel to true and when he releases that button it turns to false.

A better way to do this would be to use VoiceActivation mode and then toggle the IsMuted property on the trigger with the button. This means you won't send background noise if the button is pressed but no one is actually speaking. It also means you don't have to remember to make this change every time you update Dissonance!

[mirzasohailbaig]

Ok I switched over to using VoiceActivation mode and then toggle the IsMuted property

I'm getting about 30 fps on android device. So the app seems to send over audio but eventually it doesn't send anymore and gives the insufficient buffer warning. if you restart the app it still says insufficient warning but if you wait a little while and restart app audio sends over fine again. I attached the android log but I don't think it has any new details. Android_Log.txt

[mirzasohailbaig]

Below are logs from Editor and HoloLens.

Just to confirm what is the proper way to set Dissonance to disable record audio on hololens? I only want audio to go from android to hololens never from hololens to android.

HoloLens_UnityPlayer.log Editor_Log.txt

[martindevans]

The first log has a lot of messages like this:

Detected a frame skip, forcing capture pipeline reset (Delta Time:0.1946326)

This means that Dissonance detected a single frame taking (in this case) 194ms. My guess is the general framerate is quite low causing Dissonance to fall behind the time demands of realtime audio. Try increasing the frame size (this packs more audio into a single packet, so there's slightly less time demand). If possible, can you run the Dissonance demo scene on the device, this is a very simple scene so it should have a good framerate - if that fixes the issue then low framerate is the root cause.

I'm getting about 30 fps on android device

30 fps is just about ok (if you use large frames). But if it's dipping below 30fps occasionally that'll cause it to lag behind.

what is the proper way to set Dissonance to disable record audio on hololens

Do not create a VoiceBroadcastTrigger on the hololens, it'll never try to broadcast voice that way. You could also write a custom mic capture system (docs) which just returns null from the StartCapture method - this prevents hololens from running voice activation too (which is always running the background usually).

[martindevans]

Looking at the hololens log there's this in the log:

BasePreprocessingPipeline: Preprocessor running slow! Iteration took:309ms for 0 frames

This is very bad (I've actually not seen it happen before), it means that the CPU is very heavily loaded and the background audio encoding thread isn't getting enough time! Again, could you try this out with the demo scene to test that it's not something Dissonance is doing to overload the device.

[mirzasohailbaig]

So on both devices I now get 60 fps (on Android I just setApplication.TargetFrameRate = 60;that seems to give me 60 fps) and HoloLens already gives me 60 fps.

Which scene is it that I'm supposed to try? I've tried the HLAPI Demo and HLAPI Game World they both seem to require further configuration and I didn't find details on how to go about it in the docs.

[martindevans]

Did changing to 60fps change anything?

The HLAPI Demo is split into two scenes, HLAPI demo is a menu and HLAPI Game World is the actual scene with Dissonance in it. Add both of these scenes to your build and launch HLAPI Demo, it will launch HLAPI Game World when it needs to. There are step by step instructions in Assets/Dissonance/Integrations/UNet_HLAPI/Demo/README.txt.

[mirzasohailbaig]

(After setting 60fps on android) On initial app start gives theDetected a frame skip, forcing capture pipeline reset warning and insufficient buffer space shows up once but the audio goes over to hololens fine (tested for 10 mins worked fine).

Then I closed and restarted android app which resulted in giving me the insufficient buffer space warning multiple times and audio didn't go over (no errors in log) and on hololens got a bunch of BasePreprocessingPipeline: Lost 480 samples in the preprocessor (buffer full), injecting silence to compensate after reconnect. Even after closing and restarting both apps the audio no longer went over to hololens anymore.

Also I use Network Discovery to auto connect to clients not sure if that makes a difference.

I'll look into the demo scenes soon.

[martindevans]

On initial app start gives the ... but the audio goes over to hololens fine (tested for 10 mins worked fine).

This sounds fine, those warnings are probably just showing up because you have a bit of a dip in performance as the app is starting, that's quite common. It's sometimes worth disabling the DissonanceComms component and setting it to start up a few seconds after the scene has loaded, just to make sure it's all smooth. That's not critical though.

Then I closed and restarted android app which resulted in giving me the insufficient buffer space warning multiple times and audio didn't go over (no errors in log)

This is very odd - there shouldn't be any state carrying over from one run of the application to the next.

Could you try this out. First of all just try it to make sure it sounds as you'd expect (use headphones, otherwise you might get feedback). Then try running it twice to see if there are any errors the second time around. After that try running it after Dissonance to see if there are any issues.

hololens got a bunch of ...

It's odd that this is associated with the phone, I think this is probably a separate issue. Since you don't want to send from the hololens try implementing a null microphone as I mentioned in this comment to get rid of this warning.

[martindevans]

I use Network Discovery to auto connect to clients not sure if that makes a difference.

Shouldn't be an issue, Dissonance is pretty relaxed about networking. If you get the demo scene working that doesn't use network discovery, so that'll eliminate this as a potential issue.

[mirzasohailbaig]

Ok so I created a basicFileStreaming script which I attached to the DissonanceSetup prefab. This gives me the below warning which I want to confirm is expected, since I don't need audio transfer from hololens.

[Dissonance:Recording] (01:56:08.975) CapturePipelineManager: Failed to start microphone capture; local voice transmission will be disabled.

BasicFileStreaming.txt

[martindevans]

Yep, that warning is as expected.

[martindevans]

Did you have a chance to test out the microphone test I linked a couple of replies ago (this), or was the issue resolved?

[mirzasohailbaig]

No issue isn't resolved, just had other pending tasks.

So no audio is playing using the below script event though a microphone is added after pressing play (did it with Dissonance prefab disabled)

void Start()
    {
        AudioSource audio = GetComponent<AudioSource>();
        audio.clip = Microphone.Start(null, true, 100, 44100);
        audio.loop = true;
        while (!(Microphone.GetPosition(null) > 0)) { }
        Debug.Log("start playing... position is " + Microphone.GetPosition(null));
        audio.Play();

    }

[martindevans]

Unfortunately if that test isn't working this is a Unity issue - that script is just using the Unity mic API to directly play to a Unity AudioSource. Could you raise an issue with Unity and post a link to the issue report here?

Working around this is going to be tricky since the entire Unity Microphone API seems to be broken! Dissonance does support custom microphone systems (docs), so potentially you could write one which accesses the Android Audio API directly and reads the mic data from that.

[mirzasohailbaig]

Ignore previous comment I got it working in editor (had to open hololens project and somehow the android app in editor starts recording audio/playing audio)

Audio seems to play something I said from much earlier in mic recording session.

Also your scenes work fine in editor

[martindevans]

I got it working in editor

Was the original problem in editor, or just on the actual phone? If only on the phone, could you run that test script on the phone to see if it works.

Audio seems to play something I said from much earlier in mic recording session.

If this with that simple test script? You should hear yourself echoed back almost immediately!

Also your scenes work fine in editor

Could you also try the demo scenes out on the phone. If they work then we know the phone hardware is fine and it's just some kind of setup problem in your app that we need to work out.

[mirzasohailbaig]

I get the Insufficient buffer in the editor and the phone.

But I'll test the scenes on phone anyways if you want me to.

[martindevans]

is the issue still only after a restart of the app (as you mentioned this comment), in editor and on phone?

[mirzasohailbaig]

is the issue still only after a restart of the app (as you mentioned this comment), in editor and on phone?

yes, that was the case on the last app test I did with (60 fps) Android app but now the latest tests (detailed below) are basically the opposite of that issue.

1. Android phone (Recording Mic data) --> Editor (Playing transfered mic data)

• When testing your Demo scene no insufficient buffer on app start or after multiple restarts of android app, mic audio data transfers without issue.

2. Android phone (Recording Mic data) --> Editor [HoloLens Scene] (Playing transfered mic data)

• My scene gives the insufficient bufferonly the very first time app runs and audio is going through but all jumbled up and also saw some weird debug from dissonance on the hololens scene [image attached below].
• After that initial run the android app never seems to give the warning and just works fine (mic audio transfers fine) no warnings from hololens scene either.
• The unity's mic live voice script on android doesn't seem to continually play, will play only sometimes and seems to have lots of latency or some other issue where mic audio plays after a long time.

3. Android phone (Recording Mic data) --> HoloLens (Playing transfered mic data)

• The hololens app has none of the changes we made after I created this git Issue.
• My android app gives the insufficient bufferonly the first time app runs after install and never gives it again after an app restart.
• The unity's mic live voice script on android also doesn't seem to continually play, will play only sometimes.

Everything works pretty well just the very first mic recording session doesn't work for me. So I think I'm going to close the issue, I think my old Mic script I had attached to my network player object could have caused the issue, its the only thing that I removed. Although the script shouldn't really do anything unless I set a bool value.

MicNetwork.txt

[martindevans]

That log you show there looks like it's showing some very bad network congestion:

Large number of received packets pending dispatch (13).

This shows that the underlying network gave 13 packets to Dissonance all at once, if you're using 40ms frames that's half a second of audio that's been bunched up by bad network conditions!

Are you sending huge amounts of data over the network?

[mirzasohailbaig]

No I'm not sending huge amounts of data I only send strings, bools, ints...basic variables and only send them when needed.

Note that error only happened once out of the more than 20 times I ran it. And the audio I got on other end was also weirdly jumbled up.

[martindevans]

Note that error only happened once out of the more than 20 times I ran it

In that case I would guess it was just a brief network problem. It's also possible that it was a hitch in editor performance (interacting with the editor while the game is running can cause very bad performance problems which disrupt voice).

It sounds like the issue is mostly resolved now so I'll close this issue, don't hesitate to open another if the problem reappears :)