search

csdcorp / speech_to_text

A Flutter plugin that exposes device specific text to speech recognition capability.
BSD 3-Clause "New" or "Revised" License
381 stars 232 forks source link

Enhancement: iOS (maybe Android) attempt offline recognition #79

Closed mdvandergon closed 4 years ago

mdvandergon commented 4 years ago

Hi, thanks again for the package. I am using it for an application that has some data sensitivity and some longer transcription times. I was wondering if you could add a flag to the Flutter class to attempt offline (client-side) speech recognition. I know that this is possible on iOS, see below:

image That image is from WWDC 2019: https://developer.apple.com/videos/play/wwdc2019/256/

For Android, it seems less straightforward. A user can download the models on their device and set voice to text as "offline", but I am not sure if you can enforce it in an app like in iOS.

Thanks again!

sowens-csd commented 4 years ago

I like the idea. I'll have a look at adding it for the next release. Thanks for the input!

mdvandergon commented 4 years ago

Hi Stephen, thanks for the quick reply. Fantastic, curious to know how it goes. I wish I could be of more help here.

tsinis commented 4 years ago

Hey guys, as far as i know iOS offline recognition is working fine already. On Android not yet (at least on my Android 10 device), but for some reason i think it was working on versions 2.0.1 or later... Have to check this out.

UPDATE: So after some research I've found some interesting info:

So offline speech recognition with Stephen's @sowens-csd speech_to_text package is working fine on my Android 10 (even on actual 2.2.0 package, so I was wrong in earlier statement, sorry) out of the box, you only need to download desired offline dictionaries in Google voice keyboard settings. But it's only working with Google app version 11.9.15.21 (at least ARM version) or later, newer versions (released after May 15) are breaking offline speech recognition, in my case.

sowens-csd commented 4 years ago

Thanks for the research @tsinis I appreciate it. I've had a look at the code and it looks like you're right, currently offline is well supported but not required. For the use case where you must have offline recognition, i.e. sensitive content, there isn't yet an answer.

I see that the capability to require it is only for iOS 13+. I'm looking at adding a new onDevice flag to the listen method. Question then, my instinct would be to fail a listen call that requests onDevice if it isn't available. Would that work for your use case? You'd have to either check the OS version, or make the call with the onDevice: true flag, and then handle the error to decide what to do next. I think that's the most flexible implementation. I'll probably also add a boolean property at the LocalImageProvider level that you can use to determine whether onDevice is supported or not.

sowens-csd commented 4 years ago

There's an experimental implementation of this in the repo now for iOS. No Android support yet. Feedback welcome.

tsinis commented 4 years ago

Thanks for the research @tsinis I appreciate it. I've had a look at the code and it looks like you're right, currently offline is well supported but not required. For the use case where you must have offline recognition, i.e. sensitive content, there isn't yet an answer.

I see that the capability to require it is only for iOS 13+. I'm looking at adding a new onDevice flag to the listen method. Question then, my instinct would be to fail a listen call that requests onDevice if it isn't available. Would that work for your use case? You'd have to either check the OS version, or make the call with the onDevice: true flag, and then handle the error to decide what to do next. I think that's the most flexible implementation. I'll probably also add a boolean property at the LocalImageProvider level that you can use to determine whether onDevice is supported or not.

Thanks @sowens-csd seems like it’s a question for @mdvandergon, because I’m totally fine with offline recognition in your package, I’m glad it works at least on some devices, because I cannot test all the cases, with all possible firmware/software/brand combinations anyways — if someone will need it in my product: they will have to met a specific requirements (as OS version, Google app version, dictionaries available etc.).

mdvandergon commented 4 years ago

Thanks for looking into this further @tsinis!

And thank you, @sowens-csd. The onDevice flag and failure mode as you describe it makes perfect sense and would work for my use case. Like @tsinis was saying, I very likely will enforce iOS 13+, since on-device STT isn't restricted by a time limit Apple usually sets (battery concerns). On Android, it'd probably handle a failure by raising an alert to user, so they understand their data might be shared with Google without downloading their language dictionary.

Aside: I have a feeling (no evidence) that in iOS 14 and the next Android release, more on-device functionality will be announced and support will be increased, making this a timely release.

Thank you both for your contributions to this issue. I really appreciate it!