search

Neos-Metaverse / NeosPublic

A public issue/wiki only repository for the NeosVR project
197 stars 9 forks source link

Realtime Translation System #196

Open MR-Alex42 opened 5 years ago

MR-Alex42 commented 5 years ago

Advances in speech recognition, machine translation and speech synthesis allow to provide realtime translation capabilities in NeosVR. The system I'm supposing consists of the following components:

  1. Fast Speech recognition. Best is the Google service.
  2. Realtime and high quailty machine translation. Best one is https://www.deepl.com/en/translator.
  3. Speech synthesis in target language using a voice avatar based on the actual user voice. Best is https://lyrebird.ai/.

All three services offer APIs that allow integration so it is already possible to implement this.

People from all over the world could then communicate in their native language inside NeosVR. This will make NeosVR much more attractive for non-english speakers. The system can be further enhanced by also playing body gestures that fit to the target language.

WilsontheWolf commented 5 years ago

Mabey autotranslating UI's as well

Frooxius commented 5 years ago

This would be pretty cool. The speech recognition would also help hearing impaired users.

This will involve a some work to integrate however, so I'm not sure when we'd get to this one. Would this be a good paid feature to help cover the costs of the API's?

Simulacron3 commented 5 years ago

I've a strong interest in speech services and translation and would like to discuss the details or collaborate creating.

MR-Alex42 commented 5 years ago

I guess people would likely be willing to pay for this. Higher volumes of API calls could bring the costs per call down. You could use the voice system also to help blind people describing the scene that the user would see in words using the object names, tags and parent relationships (e. g. lamp standing on desk).

MR-Alex42 commented 5 years ago

Sharing new developments that could enable this system: https://ai.googleblog.com/2019/05/introducing-translatotron-end-to-end.html

shiftyscales commented 4 years ago

Related to #85.

This has also be implemented as a user-made tool by Anomalous using Azure speech services for voice to text translation into, and from a variety of languages. https://youtu.be/9wI3fxXxQl4