search

hecomi / uLipSync

MFCC-based LipSync plug-in for Unity using Job System and Burst Compiler
https://tips.hecomi.com/entry/2022/01/30/152519
MIT License
1.07k stars 128 forks source link

Delta calculations #44

Closed liudger closed 1 year ago

liudger commented 1 year ago

Add support for Delta calculation #25. Greatly improving the accuracy for identifying the phonemes.

Interface update and new length of mfcc array image

liudger commented 1 year ago

There are some fixes coming. Don't review yet

liudger commented 1 year ago

It should be good to go @hecomi

liudger commented 1 year ago

We could still do some optimisations for copy arrays. It now iterates over the arrays. We could use unsafe memcopy in the future

hecomi commented 1 year ago

Thank you for your pull request. I'm curious about the effectiveness of adding the Delta part to the MFCC. At present, we simply compare pre-acquired MFCCs with current ones using simple distance calculations or cosine similarity. With this, I'd like to investigate how much this additional 12 dimensions actually contribute to the accuracy. If it proves beneficial, I'd be keen on accepting your PR. However, if the impact is minimal, I would prefer to keep the project simple and may consider putting it on hold for the time being. I'd appreciate your thoughts on this.

liudger commented 1 year ago

I didn't do actual scientific test to see how much it improves. But from just using it the result looked promising. I guess we can do a comparison by using an audio file and creating sample from this file for both calculation. (it's optional the calculation) and see what is more accurate. Currently I am already on another project so this testing would be done later.

hecomi commented 1 year ago

Apologies for the delayed response. I have now reviewed your code.

There are changes that I would like to accept aside from the Delta calculation. It would be helpful if such changes could be provided in separate PRs in the future 🙇

As for the Delta calculation, I am considering accepting it as an experimental feature.

Please understand that I will be modifying your contributions to fit my coding style:

Thank you for your contributions and understanding.

hecomi commented 1 year ago

I've read through the code, and it seems to be computing a weighted average instead of performing differentiation for the calculation of Delta MFCC. Is this code working as intended? Please correct me if I'm wrong.

https://github.com/hecomi/uLipSync/pull/44/files#diff-623438b2ee4a24923c5ba18390fed6fac4f97e80a67a1543759f696a9e715a12R508-R534

liudger commented 1 year ago

Let me check again the calculation 🙈

liudger commented 1 year ago

I'll make separate pull request for the fixes and later make a new pull request for delta MFCC's