unstable head result with my own model

[lokvke]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

[lokvke]

https://github.com/ZiqiaoPeng/SyncTalk/assets/95479496/32c95529-8a80-4c9d-b07e-7045c3a4e41f

[kike-0304]

result.mp4

Do you use smooth_path?

[lokvke]

@kike-0304 just train with the default config.

[kike-0304]

use smooth_path in test may have stable head, and would it be convenient for you to share your data processing code?

[kike-0304]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

Hello, EmoTalk project does not provide data processing code. How did you generate the bs.np file?

[lokvke]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

Hello, EmoTalk project does not provide data processing code. How did you generate the bs.np file?

https://github.com/psyai-net/EmoTalk_release/blob/5179b27b2fdd1ca27fcbfa6a3264a5ecfd51d524/demo.py#L55

i don't know if it is right, maybe u can try it.

[kike-0304]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

Hello, EmoTalk project does not provide data processing code. How did you generate the bs.np file?

https://github.com/psyai-net/EmoTalk_release/blob/5179b27b2fdd1ca27fcbfa6a3264a5ecfd51d524/demo.py#L55

i don't know if it is right, maybe u can try it.

This is bs.npy obtained from audio, we may need to obtain bs.npy from video frames

[kike-0304]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

"face_rect": [ 61, 0, 384, 468 ] means [xmin, ymin, w, h]?

[jinqiupeter]

Hi @lokvke , may I ask which asr_model did you use to train your own video? if it's ave, how did you generate the aud_ave.npy file? thanks!

[lokvke]

Hi @lokvke , may I ask which asr_model did you use to train your own video? if it's ave, how did you generate the aud_ave.npy file? thanks!

just use the provided audio_visual_encoder.pth

[jinqiupeter]

Thanks, I managed to train my own video, though the result is not as good as May's

[lokvke]

Thanks, I managed to train my own video, though the result is not as good as May's

can u show your result here?

[xiaoxiongzhg]

Thanks, I managed to train my own video, though the result is not as good as May's

can u show your result here?

bs.npy is closely related to the blinking action. Will your character blink when you replace the bs.npy? In your example video, the characters do not blink.

[yangzhidao]

结果.mp4

Great, can you tell me how to get ave_npy?

[jinqiupeter]

I finally got it working:

https://github.com/ZiqiaoPeng/SyncTalk/assets/12045814/5e9e8a01-6c92-4827-b4aa-1bcb6db2499c

[jinqiupeter]

Thanks, I managed to train my own video, though the result is not as good as May's

can u show your result here?

bs.npy is closely related to the blinking action. Will your character blink when you replace the bs.npy? In your example video, the characters do not blink.

I updated code to only mask the lower part of the original video, so the characters' upper face remains the same as original video. See my sample above.

[HenryKang1]

Thanks, I managed to train my own video, though the result is not as good as May's

can u show your result here?

bs.npy is closely related to the blinking action. Will your character blink when you replace the bs.npy? In your example video, the characters do not blink.

I updated code to only mask the lower part of the original video, so the characters' upper face remains the same as original video. See my sample above.

Could you share which part do you adjust for this? It looks awesome.

[jryebread]

@jinqiupeter can you share the code adjustments you made for this? looks great.

hoping someone can create a short guide how to preprocess your own video data.

edit: welcome to join my discord server to discuss ai avatar stuff and sharing syncTalk tips https://discord.gg/jETUMmUD6h

[redstarxz]

@jinqiupeter can you share the code adjustments you made for this? looks great.

hoping someone can create a short guide how to preprocess your own video data.

edit: welcome to join my discord server to discuss ai avatar stuff and sharing syncTalk tips https://discord.gg/jETUMmUD6h

+1

[jinqiupeter]

@jinqiupeter can you share the code adjustments you made for this? looks great.

hoping someone can create a short guide how to preprocess your own video data.

edit: welcome to join my discord server to discuss ai avatar stuff and sharing syncTalk tips https://discord.gg/jETUMmUD6h

Preprocessing code is mostly the same as ER-NeRF

[HenryKang1]

ER-NeRF

What audio feature extractor did you use?

[jinqiupeter]

ave, the code is already available in nerf_triplane/provider

[hungho77]

i tried to preprocess my own video data, the bs.npy is generated from EmoTalk project.

Hello, EmoTalk project does not provide data processing code. How did you generate the bs.np file?

https://github.com/psyai-net/EmoTalk_release/blob/5179b27b2fdd1ca27fcbfa6a3264a5ecfd51d524/demo.py#L55 i don't know if it is right, maybe u can try it.

This is bs.npy obtained from audio, we may need to obtain bs.npy from video frames

hi, i use https://github.com/psyai-net/EmoTalk_release/blob/main/demo.py to create file bs.npy, but the shape output do not match with audio feature shape. Do you meet this problem, please give me advice , thanks.

[christopherohit]

Thanks, I managed to train my own video, though the result is not as good as May's

can u show your result here?

Hi can you share preprocessing code ? Thank for all your working !!!

[foxprocn]

result.mp4

@lokvke 大佬你好，请问如何训练自己的视频，你的预处理模块是否能提供下，我在自己的视频上进行训练，后续的优化效果模块与您持续共享，可以加我vx:wenfeng071555，期待与您的交流

[khaidq97]

I finally got it working:

kh_kr.mp4

Look great !! Can your share your preprocess and code

[G-force78]

@jinqiupeter can you share the code adjustments you made for this? looks great. hoping someone can create a short guide how to preprocess your own video data. edit: welcome to join my discord server to discuss ai avatar stuff and sharing syncTalk tips https://discord.gg/jETUMmUD6h

Preprocessing code is mostly the same as ER-NeRF

Amazing result. What differences?