Consistently getting NULL results while streaming

[BeanStalka]

Admittedly I have a weird implementation.

I am using the AudioStream class from your other package as a stream splitter

I have two 3rd parties at play. One handles the Wake Word and Silence Detection The other handles translating the audio to text for further processing.

When I hear the wake word, I begin writing to the stream that I feed to Bing:

//this houses the streams, fliping this flag will start copying data to the audioInputProvider.SttStream
audioInputProvider.CaptureSttAudio = true;

//task that stops the STT var triggerTask = Task.Run(() => { while (_audioInputProvider.CaptureSttAudio) { Task.Delay(30); } _logManager.Debug(nameof(SensorySpeechCaptureManager), "STT Trigger Task Has Finished"); });

_sttStreamingTask = _bingSpeechApiClient.SpeechToTextSimple(_audioInputProvider.SttStream, SensorySettings.SampleRate, triggerTask);

This all works, except when I await the _sttStreamingTask, it always returns null. Or worse it hangs.

Is there any way get some more logging or peek into why the result is null. My usual suspects (output window & Android Device Monitor) are not showing me any errors.

UPDATE: I am seeing these errors in the logcat: 09-24 17:42:54.365 E/AudioRecord-JNI(20755): Error -4 during AudioRecord native read [0:] AudioStream.Record(): readResult returned error code: -1

Any thoughts you may have would be greatly appreciated.

P.S. - I have a simple test app that uses your package and it really fast. I really just need to know how to step into the package and find the issue. So I can get the same performance as the sample app.

Thanks,

BeanStalka

[nexxuno]

Hi, you might be having the same problem I'm facing in issue #11 which wasn't solved by the fix. To debug this you have to download the project and add it to your solution and reference it instead of the nuget package.

I'm debugging on my side but not understanding much at the moment...

[BeanStalka]

Nexxuno,

I seems like it. I have pulled down the latest and am debugging now.

I will report here with my findings

wish me luck

Bean

UPDATE:

I was able to get one request to recognize before the Bing Speech services began sending back the "Service Unavailable" error message

Authentication, Maybe?

[NateRickard]

Re: the service unavailable error:

Assume this is Android... change your Android project to use the Android HttpClient/Handler. This now seems to be required to successfully make calls to the speech api and I need to update the readme to reflect that.

Re: the initial reported issue here:

That error seems to be coming from here. -1 is a generic 'Error' state, but interesting that a -4 is also being logged from the native code - I'm not sure what error -4 corresponds to.

If you look at that code, I try to handle the case where we have > 1 bad reads and just auto stop the recording. I'm not sure this has ever truly been tested since I have no way to simulate bad reads... I'd be interested to know what you find in your debugging.

[NateRickard]

Also, re: null results, the only way to really get a null response back now is if SendRequest were to return null. This should happen far less than it used to since we now propagate any exceptions and throw a few of our own. Let me know what you find.

[nexxuno]

Great find! it finally works, thanks.

For further reading: https://docs.microsoft.com/it-it/xamarin/android/app-fundamentals/http-stack?tabs=windows

"April, 2018 – Due to increased security requirements, including PCI compliance, major cloud providers and web servers are expected to stop supporting TLS versions older than 1.2. Xamarin projects created in previous versions of Visual Studio default to use older versions of TLS.

In order to ensure your apps continue to work with these servers and services, you should update your Xamarin projects with the Android HttpClient and Native TLS 1.2 settings shown below, then re-build and re-deploy your apps to your users."

[BeanStalka]

Nate,

"Assume this is Android... change your Android project to use the Android HttpClient/Handler. "

That was it alright.

I still get the task hanging once in a blue moon (1/100). When this happens my app is dead in the water.

When you throw the errors in the BingSpeechApiClient, they defiintely show up in the output, but I am left waiting indefinitely on the task i.e.

var recoResult = await _bingSttTask;<-- if an error occurs here, I am waiting forever.

Any chance we could return a Faulted status or anything really. So I can handle that and restart my wake word listening task?

Also, If you want me to put in a new issue, I will. Just let me know

FYI: currently using this:

            var sttResult = new RecognitionSpeechResult { DisplayText = "" };

            if (_sttStreamingTask.IsFaulted)
            {
                _logManager.Error(nameof(SensorySpeechCaptureManager), "STT Encountered an error.");
            }
            else
            {
                sttResult = await _sttStreamingTask;
            }

Thanks,

Bean

[nexxuno]

After more testing I have 100% good results while on wifi, and 100% errors while on 4G / LTE. I'll look further into it tomorrow (hopefully).

---

From: BeanStalka notifications@github.com Sent: Tuesday, September 25, 2018 10:00:05 PM To: NateRickard/Xamarin.Cognitive.BingSpeech Cc: Paolo Ferrazza; Comment Subject: Re: [NateRickard/Xamarin.Cognitive.BingSpeech] Consistently getting NULL results while streaming (#14)

Nate,

"Assume this is Android... change your Android project to use the Android HttpClient/Handler. "

That was it alright.

I still get the task hanging once in a blue moon (1/100). When this happens my app is dead in the water.

When you throw the errors in the BingSpeechApiClient, they defiintely show up in the output, but I am left waiting indefinitely on the task i.e.

var recoResult = await _bingSttTask;<-- if an error occurs here, I am waiting forever.

Any chance we could return a Faulted status or anything really. So I can handle that and restart my wake word listening task?

Also, If you want me to put in a new issue, I will. Just let me know

Thanks,

Bean

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNateRickard%2FXamarin.Cognitive.BingSpeech%2Fissues%2F14%23issuecomment-424480842&data=02%7C01%7C%7Cbfc875b1c09f424e563c08d623217d2c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636735024065737360&sdata=4FMNoW%2FqRy%2BElYovENBeMh%2FewvwcyJj1ApIJCNE35eE%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOxBAM_vnxHkpRP65-RKMre2W58Nlx6bks5ueotFgaJpZM4W3fHN&data=02%7C01%7C%7Cbfc875b1c09f424e563c08d623217d2c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636735024065737360&sdata=20ZnRVrEpZt7Q76P3%2Ba6oYuBtJtlSdVOHs0G2AtTuC4%3D&reserved=0.

[nexxuno]

Spoke too soon it's very hit or miss on wifi too.

---

From: BeanStalka notifications@github.com Sent: Tuesday, September 25, 2018 10:00:05 PM To: NateRickard/Xamarin.Cognitive.BingSpeech Cc: Paolo Ferrazza; Comment Subject: Re: [NateRickard/Xamarin.Cognitive.BingSpeech] Consistently getting NULL results while streaming (#14)

Nate,

"Assume this is Android... change your Android project to use the Android HttpClient/Handler. "

That was it alright.

I still get the task hanging once in a blue moon (1/100). When this happens my app is dead in the water.

When you throw the errors in the BingSpeechApiClient, they defiintely show up in the output, but I am left waiting indefinitely on the task i.e.

var recoResult = await _bingSttTask;<-- if an error occurs here, I am waiting forever.

Any chance we could return a Faulted status or anything really. So I can handle that and restart my wake word listening task?

Also, If you want me to put in a new issue, I will. Just let me know

Thanks,

Bean

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNateRickard%2FXamarin.Cognitive.BingSpeech%2Fissues%2F14%23issuecomment-424480842&data=02%7C01%7C%7Cbfc875b1c09f424e563c08d623217d2c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636735024065737360&sdata=4FMNoW%2FqRy%2BElYovENBeMh%2FewvwcyJj1ApIJCNE35eE%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOxBAM_vnxHkpRP65-RKMre2W58Nlx6bks5ueotFgaJpZM4W3fHN&data=02%7C01%7C%7Cbfc875b1c09f424e563c08d623217d2c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636735024065737360&sdata=20ZnRVrEpZt7Q76P3%2Ba6oYuBtJtlSdVOHs0G2AtTuC4%3D&reserved=0.

[nexxuno]

Had the time to test this on iOS too with the usual code you can find in #11 , I think the problem is deeper than changing the httpclient, I get the same error on iOS:

SendRequest: Non-success status received :: Status Code: ServiceUnavailable; Reason: Service Unavailable; Content: <!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>

Bing services aren't available right now

We're working to restore all services as soon as possible.
We know you want to get back to searching. Please check back soon.

Ref A: C1DE93E158A444CBB1A9B232F7B8181A Ref B: MIL30EDGE0115 Ref C: 2018-09-26T16:06:53Z

Main thread

Xamarin.Cognitive.BingSpeech
BingSpeechApiClient+<SendRequest>d__42.MoveNext ()
System.Runtime.CompilerServices
TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task)
System.Runtime.CompilerServices
TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task)
System.Runtime.CompilerServices
TaskAwaiter`1[TResult].GetResult ()
Xamarin.Cognitive.BingSpeech
BingSpeechApiClient+<SpeechToTextDetailed>d__53.MoveNext ()
System.Runtime.CompilerServices
TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task)
System.Runtime.CompilerServices
TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task)
System.Runtime.CompilerServices
TaskAwaiter`1[TResult].GetResult ()
BeltMap.Services
BingSpeechToText+<AcquireAsync>d__7.MoveNext ()

[NateRickard]

@BeanStalka re: awaiting that Task... any exception throw in the client should get surfaced, and if you're awaiting the Task returned from the SpeechToText call, it should fail and drop you into any catch block you have surrounding the call.

If I synthesize an exception in the client, the sample app has no issues completing the await and jumping to the catch block. Faulted Tasks should also raise exceptions when you await them so I'm not really sure why the code you pasted is needed, even if you keep a reference to the Task and await it later as you're doing.

The Task should be faulted strictly via the magic of using async/await and having an exception occur during the awaited code. i.e. returning a faulted status should not be a manual thing but just happen when the runtime sets the exception to the Task.

Any other info you can provide to help reproduce this? Is it possible that the triggerTask is not completing and that's causing it to never complete? Looking at the code I can see there may be an issue if your recording task never completes. Probably need to add in a timeout that matches the speech api's limit. My recording plugin auto completes after a timeout but your custom implementation may not?

[NateRickard]

And re: ServiceUnavailable error, let's keep all discussion on that in #15

[BeanStalka]

Nate,

I updated my code to use a try catch around the await of that task.

I am able to capture the error and respond accordingly.

Thanks,

Bean

[BeanStalka]

I am closing this issue since my reaming issues are around the 'Service Unavailable'