ZiqiaoPeng / SyncTalk

[CVPR 2024] This is the official source for our paper "SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis"
https://ziqiaopeng.github.io/synctalk/
Other
1.34k stars 161 forks source link

预处理中颈部的填充Cn实现 #22

Closed Kedreamix closed 8 months ago

Kedreamix commented 8 months ago

你好,我想问一下,对于预处理中的颈部的填充是如何填的呢,我借鉴于ER-NeRF的代码来进行修改

我根据论文的思想,取出对应图片的颈部的RGB颜色,然后填充,但是好像效果不一样,就请问一下,大概如何实现的 image

index = np.where(neck_part)
neck_part_image = torso_image[index[0], index[1], :]
neck_part_alpha = torso_alpha[index[0], index[1]]
mean_neck_part_color = np.mean(neck_part_image, axis=0)
mean_neck_part_alpha = np.mean(neck_part_alpha)
torso_image[inpaint_mask] = mean_neck_part_color
torso_alpha[inpaint_mask] = mean_neck_part_alpha

5_inpaint_mask

5

5

ZiqiaoPeng commented 8 months ago
    neck_avg_color = np.mean(gt_image[neck_part], axis=0)
    inpaint_neck_colors = top_neck_colors[None].repeat(L, 0)
    alpha_values = np.linspace(1, 0, L).reshape(L, 1, 1) 
    inpaint_neck_colors = inpaint_neck_colors * alpha_values + neck_avg_color * (1 - alpha_values)
    inpaint_neck_colors = inpaint_neck_colors.reshape(-1, 3)
    torso_image[tuple(inpaint_neck_coords.T)] = inpaint_neck_colors

可以参考一下这个流程。

Kedreamix commented 8 months ago

好的,非常感谢

BakingBrains commented 7 months ago

@Kedreamix can please tell how did you extract the face from the frames.

Kedreamix commented 7 months ago

you can use parsing module to extract the face