search

viromedia / viro

ViroReact: AR and VR using React Native
MIT License
2.3k stars 482 forks source link

Viseme Animation - Feature Request #338

Closed mkharibalaji closed 5 years ago

mkharibalaji commented 6 years ago

Hi Guys, Is it possible to do a viseme animation by connecting Amazon polly in future which would add wonderful AR/VR Experience helping to create something like a chatbot through AR for FBX 3d Model.

dam00n commented 6 years ago

Hi @mkharibalaji, thanks for the suggestions. We will add viseme animation to our backlog. We need to understand how this works better as it is a new concept to us. Also what did you mean by Amazon polly? We are not familiar with that product.

mkharibalaji commented 6 years ago

Hi @dam00n , Here is the way that i am expecting it to work. Imagine we have a character FBX exported from iclone (Where we usually do our Viseme Animations generally.) and the Character is the one that is going to be displayed in the AR world who would chat with the User as a Tour Guide for the Place.

1) Mobile App ViroMedia Framework. 2) When the Normal user speak to the 3d FBX character in the AR world of the Mobile App,we then convert the speech to text using Microsoft LUIS (Language Understanding Intelligent Service) for intent parsing and then we get the response from any of the chatbot frameworks through the cloud services. 3) Once we get the text back,we send the text to AWS polly which converts the text and visemes to speech in mp3 or ogg vorbis format via SSML (https://docs.aws.amazon.com/polly/latest/dg/supported-ssml.html) 4) We then get the streaming mp3 voice from AWS polly in the Mobile App which needs be processed ina way to do viseme animations on the the FBX character based on the voice streaming. (This is the Piece that i am expecting from Viromedia)

AWS Polly is a text-to speech engine on the AWS cloud that allows generating life-like speech from text.

dam00n commented 6 years ago

Hi @mkharibalaji, thanks for the detailed explanation. I have added this detail to the story in our backlog. Will update this issue when we have more info.