search

ZiqiaoPeng / SyncTalk

[CVPR 2024] This is the official source for our paper "SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis"
https://ziqiaopeng.github.io/synctalk/
Other
1.07k stars 119 forks source link

how to use ave encode？ #24

Closed kike-0304 closed 2 months ago

kike-0304 commented 4 months ago

I used ave audio encoder to replace ernerf's audio encoder, and unfortunately I got a worse result.

Elysiameik commented 4 months ago

I used ave audio encoder to replace ernerf's audio encoder, and unfortunately I got a worse result.

I have also been modifying the codes of ernerf and synctalk recently. Can you please discuss the related issues with me? my email is zcygyzytd@gmail.com

fengxiang777 commented 4 months ago

yeah,I had the same result. My guess is that the ave module is probably less effective than deepspeech. Or maybe I'm not using the ave module correctly.

I used ave audio encoder to replace ernerf's audio encoder, and unfortunately I got a worse result.

fengxiang777 commented 4 months ago

I used features from the deepspeech and ave modules as input. The result is that lse-c stays the same on the test set of Obama videos, and works much better on other audio than the deepspeech used by er-nerf.

kike-0304 commented 4 months ago

我使用 deepspeech 和 ave 模块中的功能作为输入。结果是，lse-c 在奥巴马视频的测试集上保持不变，并且在其他音频上比 er-nerf 使用的深度语音效果更好。

Great, can you tell me how you use ave?

ZiqiaoPeng commented 2 months ago

The code to get ave features has been released.