Azure-Samples / cognitive-services-speech-sdk

Sample code for the Microsoft Cognitive Services Speech SDK
MIT License
2.83k stars 1.83k forks source link

Service Timeout #453

Closed ffortunato closed 4 years ago

ffortunato commented 4 years ago

Describe the bug Hello, i've been trying to implement the samples from LUIS tutorials. What i'm trying to accomplish is to make a console application that will listen to they microphone and recognize intents. When any intent is triggered, for now, i just want to print it in the screen (basically, the same that the tutorial is doing).

I'm trying to use the ContinuousRecognition from the microphone. The issue is that after some time i will get this error: CANCELED: Reason=Error CANCELED: ErrorCode=ServiceTimeout CANCELED: ErrorDetails=Due to service inactivity the client buffer size exceeded. Resetting the buffer. SessionId: 8e431826b5c4434fb80d9f64a865bf94 CANCELED: Did you update the subscription info?

When this error appears the app will be responsive again and wait for some speech until the same error occurs.

To Reproduce Steps to reproduce the behavior: Explained above. Please let me know if i should post my code.

Expected behavior I was expecting that the program would be constantly listening and printing messages without errors.

Version of the Cognitive Services Speech SDK 1.8.0

Platform, Operating System, and Programming Language

Additional context CANCELED: Reason=Error CANCELED: ErrorCode=ServiceTimeout CANCELED: ErrorDetails=Due to service inactivity the client buffer size exceeded. Resetting the buffer. SessionId: 8e431826b5c4434fb80d9f64a865bf94 CANCELED: Did you update the subscription info?

ffortunato commented 4 years ago

`using Microsoft.CognitiveServices.Speech; using Microsoft.CognitiveServices.Speech.Intent; using System; using System.Threading.Tasks;

namespace AzureLUIS { class Program { static void Main(string[] args) { RecognizeIntentAsync().Wait(); Console.WriteLine("Please press Enter to continue."); Console.ReadLine();

    }

    static async Task RecognizeIntentAsync()
    {

        var config = SpeechConfig.FromSubscription("xxxxxxx", "westeurope");

        // Creates an intent recognizer using microphone as audio input.

        using (var recognizer = new IntentRecognizer(config))
        {
            // Creates a Language Understanding model using the app id, and adds specific intents from your model
            var model = LanguageUnderstandingModel.FromAppId("xxxxxxx");

            recognizer.AddIntent(model, "YourLanguageUnderstandingIntentName1", "id1");
            recognizer.AddIntent(model, "YourLanguageUnderstandingIntentName2", "id2");
            recognizer.AddIntent(model, "YourLanguageUnderstandingIntentName3", "any-IntentId-here");

            // The TaskCompletionSource to stop recognition.
            var stopRecognition = new TaskCompletionSource<int>();

            // Starts intent recognition, and returns after a single utterance is recognized. The end of a
            // single utterance is determined by listening for silence at the end or until a maximum of 15
            // seconds of audio is processed.  The task returns the recognition text as result. 
            // Note: Since RecognizeOnceAsync() returns only a single utterance, it is suitable only for single
            // shot recognition like command or query. 
            // For long-running multi-utterance recognition, use StartContinuousRecognitionAsync() instead.
            await recognizer.StartContinuousRecognitionAsync().ConfigureAwait(false);

            // Subscribes to events.
            recognizer.Recognizing += (s, e) =>
            {
                Console.WriteLine($"RECOGNIZING: Text={e.Result.Text}");
            };

            recognizer.Recognized += (s, e) =>
            {
                if (e.Result.Reason == ResultReason.RecognizedIntent)
                {
                    Console.WriteLine($"RECOGNIZED: Text={e.Result.Text}");
                    Console.WriteLine($"    Intent Id: {e.Result.IntentId}.");
                    Console.WriteLine($"    Language Understanding JSON: {e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult)}.");
                }
                else if (e.Result.Reason == ResultReason.RecognizedSpeech)
                {
                    Console.WriteLine($"RECOGNIZED: Text={e.Result.Text}");
                    Console.WriteLine($"    Intent not recognized.");
                }
                else if (e.Result.Reason == ResultReason.NoMatch)
                {
                    Console.WriteLine($"NOMATCH: Speech could not be recognized.");
                }
            };

            recognizer.Canceled += (s, e) =>
            {
                Console.WriteLine($"CANCELED: Reason={e.Reason}");

                if (e.Reason == CancellationReason.Error)
                {
                    Console.WriteLine($"CANCELED: ErrorCode={e.ErrorCode}");
                    Console.WriteLine($"CANCELED: ErrorDetails={e.ErrorDetails}");
                    Console.WriteLine($"CANCELED: Did you update the subscription info?");
                }

                // stopRecognition.TrySetResult(0);
            };

            recognizer.SessionStarted += (s, e) =>
            {
                Console.WriteLine("\n    Session started event.");
            };

            recognizer.SessionStopped += (s, e) =>
            {
                Console.WriteLine("\n    Session stopped event.");
                Console.WriteLine("\nStop recognition.");
                // stopRecognition.TrySetResult(0);
            };

            // Waits for completion.
            // Use Task.WaitAny to keep the task rooted.
            Task.WaitAny(new[] { stopRecognition.Task });

            // Stops recognition.
            await recognizer.StopContinuousRecognitionAsync().ConfigureAwait(false);

        }
    }
}

} `

pankopon commented 4 years ago

Thanks for reporting, and sorry about the delay in response. We will look into this and get back to you hopefully soon.

pankopon commented 4 years ago

How often do you see the error? I tested your code with just small modifications (move StartContinuousRecognitionAsync after subscribing to events, wait for key press instead of task to stop) and can run it e.g. 30+ min without errors. See the attached zip for the updated code. Program.zip

ffortunato commented 4 years ago

Hi @pankopon , thanks for the answer. The error i mentioned was recurrent. Everytime the program recognized something it would give that error and for some seconds it was stucked.

With your code it works better, however, i'm having some issue recognizing Intents.

I have created the required resource in Azure, and "imported" it in http://eu.luis.ai. Configured an Intent with some utterances and it works well when i test in http://eu.luis.ai.

It recognizes the voice, but never recognize the Intent. Do you know what's the issue?

pankopon commented 4 years ago

Have you tried the LUIS example HomeAutomation prebuilt domain, does your program recognize intents with that? This is just to verify that basic LUIS integration works.

If you haven't, see https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-get-started-create-app for how to add the domain to your app assets in the LUIS portal, then modify your console program code to add the following intents:

            recognizer.AddIntent(model, "HomeAutomation.TurnOn", "id1");
            recognizer.AddIntent(model, "HomeAutomation.TurnOff", "id2");

To test it, say things like "turn on the lights" and "turn off the lights" (or just "turn on" and "turn off").

Also please double-check the notes about keys and endpoint settings, just in case:

https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-recognize-intents-from-speech-csharp?toc=/azure/cognitive-services/luis/toc.json&bc=/azure/cognitive-services/luis/breadcrumb/toc.json

ffortunato commented 4 years ago

Hi @pankopon , thanks for your answer. Actually, i think everything is well configured.

I changed the program. ` recognizer.Recognizing += (s, e) => { Console.WriteLine($"RECOGNIZING: Text={e.Result.Text}"); };

            recognizer.Recognized += (s, e) =>
            {
                if (e.Result.Reason == ResultReason.RecognizedIntent)
                {
                    Console.WriteLine($"RECOGNIZED: Text={e.Result.Text}");
                    Console.WriteLine($"    Intent Id: {e.Result.IntentId}.");
                    Console.WriteLine($"    Language Understanding JSON: {e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult)}.");
                }
                else if (e.Result.Reason == ResultReason.RecognizedSpeech)
                {
                    Console.WriteLine($"RECOGNIZED: Text={e.Result.Text}");
                    Console.WriteLine($"    Intent not recognized.");
                    var client = new WebClient();
                    var content = client.DownloadString($"https://luissapphire.cognitiveservices.azure.com/luis/prediction/v3.0/apps/fac600d9-ecfb-459b-9ac7-43481481dd86/slots/production/predict?subscription-key=cd5e0f3e8bb347faa055c9b172ddd413&verbose=true&show-all-intents=true&log=true&query={e.Result.Text}");
                    Console.WriteLine(content);
                }
                else if (e.Result.Reason == ResultReason.NoMatch)
                {
                    Console.WriteLine($"NOMATCH: Speech could not be recognized.");
                }
            };`

As you can see, if i call the endpoint using a WebClient everything is ok and i receive a JSON as response with the percentage for each of my Intents.

My expectation was that i didn't had to do any of this and that the C# Library would take care of it. The result would appear when e.Result.Reason == ResultReason.RecognizedIntent.

The subscriptionKey and appId that i used in the URL endpoint are the same that i used in the code.

But as suggested i just added the the prebuilt HomeAutomation but the same seems to be happening.

What can i be doing wrong ?

pankopon commented 4 years ago

In case of RecognizedSpeech, you can also read a JSON result directly from e.Result in the same way as with RecognizedIntent. Using the HomeAutomation domain, the output would be like follows when the speech input is "What's the weather like?" i.e. not a recognized intent

RECOGNIZING: Text=what RECOGNIZING: Text=what's RECOGNIZING: Text=what's the RECOGNIZING: Text=what's the weather RECOGNIZING: Text=what's the weather like RECOGNIZED: Text=What's the weather like? Intent not recognized. Language Understanding JSON: { "query": "what's the weather like", "topScoringIntent": { "intent": "None", "score": 0.0785802156 }, "entities": [] }.

For a recognized intent it would be e.g.

RECOGNIZING: Text=turn RECOGNIZING: Text=turn off RECOGNIZING: Text=turn off the RECOGNIZING: Text=turn off the light RECOGNIZING: Text=turn off the lights RECOGNIZED: Text=Turn off the lights. Intent Id: id2. Language Understanding JSON: { "query": "turn off the lights", "topScoringIntent": { "intent": "HomeAutomation.TurnOff", "score": 0.233853281 }, "entities": [] }.

I've attached an updated program code which uses HomeAutomation domain and prints the LUIS JSON in both speech and intent cases, can you please verify that your program does the same, and copy the output here? Program2.zip

Also, just to make sure, do you have a paid LUIS subscription, as the free one does not support speech requests for intent recognition?

ffortunato commented 4 years ago

Hi @pankopon , i was using the free one. That was the issue :)

pankopon commented 4 years ago

Ok, good to know that the reason was found.

json0755 commented 2 years ago

CANCELED: Reason=Error CANCELED: ErrorCode=ServiceTimeout CANCELED: ErrorDetails=Timeout while synthesizing. USP state: 4. Received audio size: 36870 bytes. CANCELED: Did you set the speech resource key and region values?

Who can help me?

json0755 commented 2 years ago

com.microsoft.cognitiveservices.speech.SpeechSynthesizer speechSynthesizer = new com.microsoft.cognitiveservices.speech.SpeechSynthesizer(speechConfig,null);

SpeechSynthesisResult result = speechSynthesizer.SpeakTextAsync("Dear babies, store membership can be 0.01 yuan for the first time to take a cheese article squid? The host told everyone at the same time, we shop member first order is a new member can enjoy a reduction in cash, but not every link have oh, everybody can opens at the lower left our baby a pocket to view in detail.").get();

Console: CANCELED: Reason=Error CANCELED: ErrorCode=ServiceTimeout CANCELED: ErrorDetails=Timeout while synthesizing. USP state: 4. Received audio size: 36870 bytes. CANCELED: Did you set the speech resource key and region values?

Who can help me?

tianyi14 commented 1 year ago

com.microsoft.cognitiveservices.speech.SpeechSynthesizer speechSynthesizer = new com.microsoft.cognitiveservices.speech.SpeechSynthesizer(speechConfig,null);

SpeechSynthesisResult result = speechSynthesizer.SpeakTextAsync(“亲爱的宝宝们,店内会员可以0.01元第一次吃奶酪文章鱿鱼吗?主持人同时告诉大家,我们店员第一单是新会员可以享受减金,但不是每个环节都有哦,大家可以在左下角打开我们宝宝一个口袋详细查看。获取();

控制台:已取消:原因=错误已取消:错误代码=服务超时已取消:错误详细信息=合成时超时。USP 状态:4。 收到的音频大小:36870 字节。已取消:是否设置了语音资源键和区域值?

谁能帮助我? 你好,老哥,这个问题解决了吗,是什么原因导致的