SpeechClient.dll frequently crashes

[leviyakir]

Recently we have upgraded version 0.4.10.1 to 1.0.0.2 (the current one) hoping it will be more stable. With the old version we've had 2 crashes: 1) When going online after being offline. 2) When deleting the DataRecognitionClient object after calling EndAudio, because callbacks were still streaming results (we use managed C++).

With the new version we have 2 crashes of which the first is catastrophic: 1) Very frequently: Less than a seconds after sending audio to the transcription server SpeechClient.dll working thread crashes. 2) As in #2 above.

Stack traces for the new crashes: 1) ExceptionAddressSymbolInfo=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D6783, module offset(base): 0x46783(0x7ffc0f290000) StackTrace00=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D6783, module offset(base): 0x46783(0x7ffc0f290000) StackTrace01=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D744E, module offset(base): 0x4744e(0x7ffc0f290000) StackTrace02=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D8A95, module offset(base): 0x48a95(0x7ffc0f290000) StackTrace03=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D7DD9, module offset(base): 0x47dd9(0x7ffc0f290000) StackTrace04=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D3A8D, module offset(base): 0x43a8d(0x7ffc0f290000) StackTrace05=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2D31C8, module offset(base): 0x431c8(0x7ffc0f290000) StackTrace06=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2C18DA, module offset(base): 0x318da(0x7ffc0f290000) StackTrace07=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2FF087, module offset(base): 0x6f087(0x7ffc0f290000) StackTrace08=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFC0F2FEC4D, module offset(base): 0x6ec4d(0x7ffc0f290000)

2) ExceptionAddressSymbolInfo=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4E30209, module offset(base): 0x60209(0x7ffdc4dd0000) StackTrace00=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4E30209, module offset(base): 0x60209(0x7ffdc4dd0000) StackTrace01=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4E30398, module offset(base): 0x60398(0x7ffdc4dd0000) StackTrace02=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4DFDE90, module offset(base): 0x2de90(0x7ffdc4dd0000) StackTrace03=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4DFF499, module offset(base): 0x2f499(0x7ffdc4dd0000) StackTrace04=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4E3F04C, module offset(base): 0x6f04c(0x7ffdc4dd0000) StackTrace05=C:\Program Files\Screenovate\BluePhone\SpeechClient.dll! DllUnregisterServer - 0x00007FFDC4E3EC4D, module offset(base): 0x6ec4d(0x7ffdc4dd0000)

[xin9le]

A same problem has occurred in my environment, and I'm really confused because exception can NOT be caught. This speech to text API is extremely wonderful and a necessary function for us. I hope it'll be fixed as soon as possible.

[xin9le]

Following is simple reproduction code. Please check it out.

Reproduction code

using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.CognitiveServices.SpeechRecognition;

namespace BingSpeechApi
{
    class Program
    {
        static void Main()
        {
            try
            {
                var tasks = Enumerable.Range(0, 100).Select(_ => AnalyzeAsync());
                Task.WhenAll(tasks).Wait();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
            Console.ReadLine();
        }

        private static async Task AnalyzeAsync()
        {
            using (var client = SpeechRecognitionServiceFactory.CreateDataClient(SpeechRecognitionMode.ShortPhrase, "en-US", "API key here"))
            {
                //--- callback
                var tcs = new TaskCompletionSource<bool>();
                client.OnResponseReceived += (sender, e) =>
                {
                    foreach (var x in e.PhraseResponse.Results)
                        Console.WriteLine(x.DisplayText);
                    tcs.SetResult(true);
                };
                client.OnPartialResponseReceived += (sender, e) =>
                {
                    Console.WriteLine(e.PartialResult);
                };

                //--- send data
                using (var fs = new FileStream("WAVE file path here", FileMode.Open, FileAccess.Read))
                {
                    var buffer = new byte[1024 * 10];
                    var readBytes = 0;
                    do
                    {
                        readBytes = await fs.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
                        client.SendAudio(buffer, buffer.Length);
                    }
                    while (readBytes > 0);
                }
                client.EndAudio();

                //--- wait for result 
                GC.Collect();  // In order to make error easily.
                await tcs.Task;
            }
        }
    }
}

Occured exception

Type

System.AccessViolationException

Source

SpeechClient

Message

保護されているメモリに読み取りまたは書き込み操作を行おうとしました。他のメモリが壊れていることが考えられます。

StackTrace

場所 HostCallbacks.{ctor}(HostCallbacks* , INameValueCollection* , IntPtr , IntPtr , IntPtr , IntPtr , IntPtr )
場所 Microsoft.CognitiveServices.SpeechRecognition.Conversation.InitializeBase(Preferences prefs)
場所 Microsoft.CognitiveServices.SpeechRecognition.Conversation.CreateAudioStream(SpeechAudioFormat format)
場所 Microsoft.CognitiveServices.SpeechRecognition.DataRecognitionClient.SleepAndSendAudioWorker()
場所 Microsoft.CognitiveServices.SpeechRecognition.WorkQueue.ThreadWorker()
場所 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
場所 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
場所 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
場所 System.Threading.ThreadHelper.ThreadStart()

[priyaravi20]

Sorry for the inconvenience. Bing Speech team is aware of the issue and actively working to resolve it. I will provide an update before EOD.

Thanks Priya Rajan, Bing Speech PM

[jasonvomsft]

@leviyakir your exceptions mentioned for 1.0.0.2 seem to point to your machine not having a Microphone. Can you confirm?

@xin9le I can't reproduce with your sample. Can you try the public sample and verify that works?

@leviyakir and @xin9le if you have new exceptions, please provide crash dumps..

[xin9le]

Thank you for the quick answer :)

I tried your public sample, I could NOT reproduce.However I think the reason why it's working on single thread. I wanna use this SDK on multi-thread environment (IOW server side). That's the reason why, I presented a sample that works on multi-thread.

I wonder if the combination of thread management and native function call is not good. Especially, I think this problem occurs around the DataRecognitionClient.Dispose().

Following attachment file is one of the crash report. I hope it helps you.

• Report.zip

Best regards,

[xin9le]

If thread management is the cause, the following way may also be effective for reproduction. Please try this.

static void Main()
{
    try
    {
        //--- add these
        ThreadPool.SetMinThreads(1, 1);
        ThreadPool.SetMaxThreads(2, 2);

        var tasks = Enumerable.Range(0, 100).Select(_ => AnalyzeAsync());
        Task.WhenAll(tasks).Wait();
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex);
    }
    Console.ReadLine();
}

[leviyakir]

My machine has a microphone. Regardless, I am using the data client feature and sending audio buffers. I have 2 dumps for case 1: https://drive.google.com/file/d/0BwtlfDyTIqVmVzFtSmlHbFRwaVU/view?usp=sharing and https://drive.google.com/file/d/0BwtlfDyTIqVmbUVULTVKTW5fZ0E/view?usp=sharing I will look for dumps for case 2.

[leviyakir]

There is case 3 I haven't mentioned yet: Start transcribing when not connected to Internet and then connect. 100% repro of crash. I attached a dump and a revised version of the published sample that you can reproduce this crash with. I used the wav file for long dictation mode. Link to dump, exe, pdb, and source files (to replace existing ones): https://drive.google.com/file/d/0BwtlfDyTIqVmMFpna3ZaUUp6eXc/view?usp=sharing Call stack:

KERNELBASE.dll!DebugBreak() Unknown websocket.dll!ReportFatalException(unsigned int64,unsigned long,enum FATAL_FAILURE_REASON_TYPE,unsigned long) Unknown websocket.dll!WebSocket::VerifyCookie(void) Unknown websocket.dll!WebSocketReceive() Unknown winhttp.dll!WebSocketLibrary::Receive(union _WEB_SOCKET_BUFFER *) Unknown winhttp.dll!WEB_SOCKET_HANDLE_OBJECT::BeginBackgroundReceive(void) Unknown winhttp.dll!WEB_SOCKET_HANDLE_OBJECT::Initialize(unsigned int64) Unknown winhttp.dll!WEB_SOCKET_HANDLE_OBJECT::WEB_SOCKET_HANDLE_OBJECT(class INTERNET_CONNECT_HANDLE_OBJECT ,int,void ()(void ,unsigned __int64,unsigned long,void ,unsigned long),unsigned long,unsigned int64,class WebSocketLibrary *,unsigned int64) Unknown winhttp.dll!RMakeWebSocketObjectHandle(class INTERNET_CONNECT_HANDLE_OBJECT ,int,void ()(void ,unsigned __int64,unsigned long,void ,unsigned long),unsigned long,unsigned int64,class WebSocketLibrary *,unsigned int64,class WEB_SOCKET_HANDLE_OBJECT ) Unknown winhttp.dll!WinHttpWebSocketCompleteUpgradeCore(void ,unsigned __int64,void ) Unknown winhttp.dll!WinHttpWebSocketCompleteUpgradeInternal(void ,unsigned __int64) Unknown winhttp.dll!WinHttpWebSocketCompleteUpgrade() Unknown SpeechClient.dll!00007ffcdf08c515() Unknown SpeechClient.dll!00007ffcdf08c9e4() Unknown SpeechClient.dll!00007ffcdf08bf65() Unknown winhttp.dll!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback() Unknown winhttp.dll!WinHttpReceiveResponse() Unknown SpeechClient.dll!00007ffcdf08cd46() Unknown SpeechClient.dll!00007ffcdf08c796() Unknown SpeechClient.dll!00007ffcdf08ca13() Unknown SpeechClient.dll!00007ffcdf08bf65() Unknown winhttp.dll!HTTP_USER_REQUEST::OnSendRequest() Unknown winhttp.dll!WEBIO_REQUEST::OnIoComplete() Unknown winhttp.dll!HTTP_THREAD_POOL::_StaticWorkItemCallback() Unknown ntdll.dll!TppWorkpExecuteCallback() Unknown ntdll.dll!TppWorkerThread() Unknown kernel32.dll!00007ffd3ddf8364() Unknown ntdll.dll!RtlUserThreadStart() Unknown

[leviyakir]

3 more dumps for case 1: https://drive.google.com/open?id=0BwtlfDyTIqVma3ZFUENWWGdIWVU (same call stack). And another (a different call stack): https://drive.google.com/file/d/0BwtlfDyTIqVmbl9pV01EdUE3LUE/view?usp=sharing

[leviyakir]

Dumps for case 2 (when stopping). This one is surely for stop: https://drive.google.com/file/d/0BwtlfDyTIqVmTlU3amRoSi1VMVU/view?usp=sharing This one is different and probably for stop as well: https://drive.google.com/file/d/0BwtlfDyTIqVmcnNBdkJtX1czWm8/view?usp=sharing

[leviyakir]

Another one for case 1: https://drive.google.com/file/d/0BwtlfDyTIqVmc2h2RzlzWkF4dnc/view?usp=sharing

[xin9le]

@priyaravi20 @jasonvomsft What did you come up with to solve this problem? Can you tell me about the progress on this?

Best regards,

[RayGrooms]

We are unable to duplicate the issue with the latest version of SpeecClient.dll that i have downloaded from nuget(1.0.0.3).

[Plasma]

I'm receiving this too. If I Dispose() of the client some time later I get the exception that cannot be caught by any .NET code.

[Plasma]

This is in the latest 1.0.3 client btw

[imzaheermehmood]

Its crashing on windows 10 machines. I am using latest 1.0.0.6.

[priyaravi20]

Hi - Since its a long thread I want to understand the exact steps you use to repro so we can dig deeper. I have a windows 10 machine and have not been able to repro so it will be a huge help if you can provide us more info. Thanks!

[imzaheermehmood]

Thank you for your reply. I had other Redistributables but Microsoft Visual C++ 2015 Redistributable was missing on some machines. It fixed my issue.