search

virtual-puppet-project / vpuppr

VTuber application made with Godot 4
Mozilla Public License 2.0
741 stars 61 forks source link

Lip sync support #83

Closed you-win closed 1 year ago

you-win commented 2 years ago

Add lip sync support. This will likely need to be done as a GDNative library for performance reasons. Rust will probably be the implementation language.

uLipSync (MIT) can be used as a reference even though it's a Unity library.

godot-audio-processing Can be used as a reference for feeding data from Godot and maybe interpreting the lip synced data?

NOTE: There's no license for godot-audio-processing, but I did ask on Discord if I could use the repo as a reference

Capture

personal fork of uLipSync

personal fork of godot-audio-processing

GeorgeS2019 commented 2 years ago

@you-win

Would you be interested Text to Speech LipSync?

you-win commented 2 years ago

@GeorgeS2019 looks like #90 will be handling your request, as your request is more about TTS than lip syncing. This issue is tracking only lip sync support.

On that note, #88 implements basic lip sync support. I will need to add phoneme (viseme?) detection in a later PR.