search

csdcorp / speech_to_text

A Flutter plugin that exposes device specific text to speech recognition capability.
BSD 3-Clause "New" or "Revised" License
351 stars 218 forks source link

How to know when plugin is done listening and has a result #50

Closed ElZombieIsra closed 4 years ago

ElZombieIsra commented 4 years ago

I'm aware that you can pass the onResultfunction in the listen method to the plugin. But that function is executed every time the plugin recognize words.

The thing that I want to do is execute a function when the plugin is all done with listening and recognizing. In fact, when the plugin reproduces the done audio in Android would be the perfect moment to execute a function.

Is there a way of doing this that I'm not aware of?

If that's possible, I think it would be helpful add it to the README for future reference to anyone with the same problem that I have.

Thanks

sowens-csd commented 4 years ago

Yes, you can do that using the partialResults flag on the listen method. It defaults to true, set it to false to only receive the final results. See the documentation on the listen method for details. 

speechToText.listen( onResult: resultCallback, partialResults: false )
ElZombieIsra commented 4 years ago

Thanks. I didn't know this param existed because I was using an older version. This seems to solve my problem, thanks.

Although the onResult function seems to be triggering twice when the plugin recognize words. You can even hear the result audio in android playing twice.

At first I thought i was triggering twice the "listen" function but I checked my logs and doesn't seem so.

sowens-csd commented 4 years ago

I'm not sure what you mean by "You can even hear the result audio in android playing twice."? The plugin doesn't play audio, it just listens. I'll check for duplicate invocation of the onResult.

ElZombieIsra commented 4 years ago

There's a default audio playing in Android every time you listen for speech. It's in the README too. You have a section where you can set an audio to play in iOS.

sowens-csd commented 4 years ago

Interesting, it does seem there's something odd going on with the onResult callback. Checking now.

ElZombieIsra commented 4 years ago

Thanks. Also, if it's not a lot to ask. You think you can add an onFinished callback or something that executes when the plug-in is done besides the onResult callback?

sowens-csd commented 4 years ago

The issue seems to only happen in Android. The duplicate onResult calls seem to be happening only when Android is auto-terminating the listen session due to a pause. In the example app if you can click the stop button before the pause detection shuts it down then you get one callback. From what I can tell so far Android is sending a duplicate notification in that case. I'm trying to figure out how to detect and block the duplicate.

sowens-csd commented 4 years ago

When would you want this new onFinished to be called? What do you consider the 'plugin is done' state?

ElZombieIsra commented 4 years ago

I think this could be executed when the plug-in has recognized words but it is done listening. Like when the done audio is reproduced in Android. That way you can still get the partial results of the recognized speech with the onResult callback and, say, print them in the UI, but you can execute another callback when there's no more speech to recognize since the plug-in is not listening anymore. And you can, say so, make an http call to your service with the final recognized speech.

I think this would add a lot of value to this plug-in since it would be a lot easier to use and implement than having to write that logic by yourself every time you want to use this.

sowens-csd commented 4 years ago

I think the finalResult property of the SpeechRecognitionResult does that. If that is true, then the recognition is done and you have the final results. The recognizer is no longer listening nor will it return further results until a new listen session is triggered. At that point you could trigger an http call or do whatever else you'd like with the results.

The bug with duplicate onResult notifications causes a problem with that, but I'll resolve that issue.

ElZombieIsra commented 4 years ago

Ok. Let me check out it's behaviour and see if that's useful to me.

Thanks

sowens-csd commented 4 years ago

New version in the repo now. It should resolve the duplicate notifications.

yoali11 commented 4 years ago

New version in the repo now. It should resolve the duplicate notifications.

FYI the new version didn't fix the duplicate onResults call. This bug is from within Android SpeechRecognizer. I'm experiencing it myself on my own SpeechRecognition plugin.

sowens-csd commented 4 years ago

I agree that it's coming from within the Android part of the plugin, that's where I put the fix. You say "...on my own SpeechRecognition plugin...' and I'm not quite sure what you meant by that? Are you using the latest version of Speech_to_text plugin and seeing duplicates? Or have you written your own plugin for SpeechRecognition and are seeing a similar problem? Could you describe the behaviour you are seeing? What OS and device type? Are you getting partial results or final results? You'll potentially see duplicates for partial as I didn't try to address that.

There are possible problems with the fix since it is really more of a work around. The issue seemed to me to be that the Android callback is getting triggered multiple times for the same recognition task. To resolve it I don't fire a notification if the trigger happens < 100 ms apart. It is possible on a slow device that the duplicates could come further apart than that. If you turn on debugLogging on the initialize call then there's a log message every time a duplicate is dropped. If you're still seeing the duplicates issue in this plugin then I'll add some logging and ask you to try running it to see what the behaviour is on your device.

ElZombieIsra commented 4 years ago

I think the finalResult property of the SpeechRecognitionResult does that.

You were right about this statement. That works perfectly for me. Thanks.

I will close this issue because it has been resolved. I think the issue about the double call in android may be treated in a different issue unless you think otherwise.

yoali11 commented 4 years ago

I agree that it's coming from within the Android part of the plugin, that's where I put the fix. You say "...on my own SpeechRecognition plugin...' and I'm not quite sure what you meant by that? Are you using the latest version of Speech_to_text plugin and seeing duplicates? Or have you written your own plugin for SpeechRecognition and are seeing a similar problem? Could you describe the behaviour you are seeing? What OS and device type? Are you getting partial results or final results? You'll potentially see duplicates for partial as I didn't try to address that.

I am using the android.speech.SpeechRecognizer API in my own custom Flutter plugin and I am too experiencing the duplicate call of onResult. Only on final results. I am pretty sure this bug is within the Android API and not your (or my own) plugin.

sowens-csd commented 4 years ago

Ah, thanks for letting me know! I agree with you, that does seem to be what's happening. Strangely I haven't seen any discussion of this problem. If you're curious have a look at the work around I put into the Android plugin logic to work around the issue. It might help with your use case.

yoali11 commented 4 years ago

Thanks. Although the fix won't remove the annoying duplicate stop-sound. It seems to be that Google don't prioritise fixing issues in SpeechRecognizer since they launched their payed version, Google Cloud Speech API...