search

Wendy-Nam / RenPy-Lipsync-Plugin

Seamless lip-sync animations for Ren'Py using Rhubarb Lip Sync
MIT License
15 stars 2 forks source link

[Update Plan] Planned Bug fix & updates on Ren'Py Lip-sync Tool #8

Open Wendy-Nam opened 1 week ago

Wendy-Nam commented 1 week ago

I'm preparing a major update for the program. Previously, I couldn't address the issues because my skills weren't up to par... (+ due to busy academic schedules)

Here are my main priorities:

  1. Advanced integration with Ren'Py’s native code (top priority)
  2. Simplifying usage and improving the interface
  3. Automating processes
  4. Lip-sync generation from text without voice files (with adjustable speed)

My first focus is on Ren'Py integration. Once that's resolved, I'll move on to items 2, 3, and 4.

The current lip-sync system is basic, relying on screen pauses and lip changes, which leads to timing issues and interaction problems. I applied quick fixes, but they were superficial and didn't solve the root problem.

To truly fix this, I needed to understand Ren'Py’s core structure, including internal functions like say, interaction, and context. Recently, through other development work, I've gained more knowledge of these areas and feel ready to implement a better solution.

A Personal Note

Although game development is just a hobby, I’m committed to improving this tool. The support from users — from some feedback to donations — All of them was new to me and has been incredibly meaningful.

I'll begin testing in November-December, with full improvements planned by January-March. Thank you for your patience, and stay tuned for updates!

Jaybe-Games commented 1 week ago

I've found a way to make LipSync super accurate, but it's still manual and not automatic. instead of doing anything with RenPy's core, I just generate an image as a sequence of mouth shapes, like an animation, that I can play in sync with a voice. I have a py script that doesn't run in RenPy, it takes a voice file and scans the volume every 50ms and depending on how loud the chunk is, the mouth is shown open or closed. This looks much better than working with sounds like A E I O U, especially if you only have three mouth shapes.

Wendy-Nam commented 1 week ago

Very interesting. If you don't mind asking, can you share me the library or code?

Jaybe-Games commented 1 week ago

I sent you the script on discord, check it out