search

niochat / nio

💬 Nio is an upcoming matrix client for iOS.
https://niochat.github.io/
Mozilla Public License 2.0
542 stars 43 forks source link

Transcribe voice messages #147

Open kiliankoe opened 4 years ago

kiliankoe commented 4 years ago

On receiving a voice message Nio could show a button that would use iOS' voice recognition APIs to transcribe it to text, similar to how it's done for voicemail. Recognition of the language would possibly be tricky though 🤔 Is there already an API that can guess a language from text which could be fed a few textual messages for context?

Related to https://github.com/turt2live/matrix-wishlist/issues/362

regexident commented 4 years ago

I'm not aware of an API to detect language in spoken text. We could however make use of the fact that people are unlikely to switch language mid-conversation from a message-to-message basis. So we could perform speech recognition with —say— the top 3 dominant languages detected in the text messages in the room and then either store them all, or pick the one with the highest confidence value.

When using SFSpeechRecognizer we should also make sure to only perform speech recognition if .supportsOnDeviceRecognition == true with .requiresOnDeviceRecognition = true, to avoid leaking sensitive information from the user's device to a remote server.

kiliankoe commented 4 years ago

That's exactly what I meant by feeding it textual messages 👍 And thanks for the hint to requiresOnDeviceRecognition, that's definitely a very sensible option here!

regexident commented 4 years ago

Lol, I somehow skipped over "be fed a few textual messages for context?". 🙈 But yeah, this should work. Alternatively the user's keyboard locale could be used as a hint.
And as far as I remember SFSpeechRecognizer actually uses it to guess the locale for the default .init().

kiliankoe commented 4 years ago

And it's always possible to present the user with a list of languages to choose from to either correct the choice made by the app or if the app determines that a confidence threshold can not be hit.