search

open-source-ideas / ideas

💡 Looking for inspiration for your next open source project? Or perhaps you've got a brilliant idea you can't wait to share with others? Open Source Ideas is a community built specifically for this! 👋
6.58k stars 221 forks source link

IPA (International Phonetic Alphabet) to mouth animation #58

Closed FurkanKambay closed 1 year ago

FurkanKambay commented 6 years ago

I don't know if this is worthwhile but it was in my "Draft Ideas" note page so here goes nothing.

This is for video games when characters talk. Their mouths will move according to IPA so it will look realistic.

These will probably give you an idea of what's in my head:

detailed

vowels

So what I'm imagining is like this:

Dialog: Hello there Translated into IPA: həˈloʊ ðɛr (we haven't used this tool I'm talking about yet) Translated into animation of the character's mouth with the tool.

I suppose only the lips' and tongue's animation would be enough.

Do you think this is stupid? Is it already done or being worked on? Or if you need more detail, let me know.

KOLANICH commented 6 years ago

I guess it may be better to transform speech sound into animation. http://www.yisongyue.com/publications/siggraph2017_speech.pdf

jleeothon commented 6 years ago

Like you mentioned, only the part about lips and tongue would be visible in the case of a videogame.

Otherwise, for scholarly purposes I think it's a very useful idea to represent these in an animated diagram. When studying phonetics, I've seen courses where the course material includes animations--no idea where they got them from.

Apart from that, and for scholarly / fun purposes, there's this app:

https://dood.al/pinktrombone/

naturallymitchell commented 6 years ago

http://soundsofspeech.uiowa.edu/index.html#english https://www.youtube.com/watch?v=bOMb0t7I93g https://www.youtube.com/watch?v=XebRQXfsTFI https://www.youtube.com/watch?v=16b2M-YwgKs

naturallymitchell commented 6 years ago

https://www.youtube.com/watch?v=dfoRdKuPF9I https://www.youtube.com/watch?v=u7jQ8FELbIo

KOLANICH commented 5 years ago

https://github.com/imatge-upc/speech2face

FurkanKambay commented 4 years ago

well, nvidia did it

naturallymitchell commented 4 years ago

Link?

FurkanKambay commented 4 years ago

@naturallymitchell It's a part of their new "Omniverse Machinima" tool which isn't available right now but you can register for beta here: https://www.nvidia.com/en-us/geforce/news/omniverse-machinima/

They're calling the tech Audio2Face (which I guess is not the same thing with what I suggested; I might've jumped the gun while watching the live stream - but I guess this way might be even better since not everyone speaks the exact same way – accents and such – and those differences should be reflected in the mouth animations)