search

HowieMa / CVTHead

[WACV 2024] "CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer"
https://arxiv.org/pdf/2311.06443.pdf
96 stars 8 forks source link

edit headpose only and keep the expression #5

Open Farewell-ME opened 3 weeks ago

Farewell-ME commented 3 weeks ago

hi there, great project! in the inference.py --flame mode, in the demo that edits exp, it seems that the exp coef is randomly given rather than keep the source image expression for i in range(10): exp = torch.zeros(1, 100).to(src_img.device) # (1, 100) exp[0, 0] = 2 * i / 10 Is there any way to edit the head pose only and keep the input face expression the same?

HowieMa commented 3 weeks ago

You can keep the exp as a None value. i.e., if you don't pass the exp value, it will follow the original expression coefficient. The logic is here: https://github.com/HowieMa/CVTHead/blob/8f75b227fe35d3aec30920064ef7c9df5dcd8567/models/cvthead.py#L436

Farewell-ME commented 2 weeks ago

i see, then the drv_codedict['exp'] will keep the same as： src_codedict = self.deca.encode(crop_src_img, use_detail=True) however, when I input './examples/2.png' and print its src_codedict['exp'] the 50-dims FLAME exp coeff, it is very close to zeors like tensor([[ 0.6193, -0.2783, 0.2837, 0.3630, -0.0149, -0.3751, -0.2054, 0.2826, -0.1413, 0.4739, 0.0610, 0.7318, 0.3283, -0.2038, 0.0750, -0.2088, -0.2746, -0.0101, 0.1676, 0.3696, -0.0183, -0.1984, -0.0575, -0.1980, -0.0062, -0.0128, 0.1302, 0.0862, -0.1784, 0.0785, -0.1404, 0.1140, 0.0795, -0.0530, 0.0306, -0.0548, 0.0012, -0.0391, 0.0938, -0.1831, -0.4531, 0.1092, -0.0538, -0.0640, -0.1024, -0.0648, 0.0657, 0.1505, 0.1055, 0.0444]], device='cuda:0') and the output pose.gif is close to neutral face and not consistent with the source image pose

is there any tricks in deca.encode and deca.decode?

HowieMa commented 2 weeks ago

Oh you also need to keep the original jaw pose. In the flame model, the model is always closed if you keep the default jaw pose.