Novel pose animation

[fluide1022]

When running the novel pose animation: python demo/test_ani.py configs/ssdnerf_avatar_uncond_thuman_conv_16bit.py work_dirs/ssdnerf_avatar_uncond_16bit_thuman_conv/ckpt/ssdnerf_avatar_uncond_16bit_thuman_conv/latest.pth --gpu-ids 0 It report error: ~/E3Gen/lib/apis/test.py", line 238, in animate_3d data['cond_poses'] = data['cond_poses'][:, 31:32] KeyError: 'cond_poses'

[olivia23333]

You can try changing the key "cond_poses" to "test_poses", the rendered result needs to be flipped horizontally and vertically to get the final animation results. If it still doesn't work, please update the error below, I will check it again. Sorry for the inconvenience.

[fluide1022]

1. load your checkpoint error: unexpected key in source state_dict: decoder.init_pcd, decoder.init_rot, decoder.face_mask, decoder.hands_mask, decoder.outside_mask, decoder.deformer.init_spdir, decoder.deformer.init_podir, decoder.deformer.init_faces, decoder.deformer.init_lbsw, decoder.deformer.tfs_inv_t, decoder.deformer.vs_template, decoder.deformer.smpl_faces, decoder.deformer.pose_offset_cano, decoder_ema.init_pcd, decoder_ema.init_rot, decoder_ema.face_mask, decoder_ema.hands_mask, decoder_ema.outside_mask, decoder_ema.deformer.init_spdir, decoder_ema.deformer.init_podir, decoder_ema.deformer.init_faces, decoder_ema.deformer.init_lbsw, decoder_ema.deformer.tfs_inv_t, decoder_ema.deformer.vs_template, decoder_ema.deformer.smpl_faces, decoder_ema.deformer.pose_offset_cano. but it does not influence the code running.
2. I change the line 238 to 241 in test.py: data['test_poses'] = data['test_poses'][:, 31:32] data['test_intrinsics'] = data['test_intrinsics'][:, 31:32] betas = data['test_smpl_param'][:, 70:80].unsqueeze(1).expand(-1, 50, -1).float() data['test_smpl_param'] = torch.cat((smplx_param.unsqueeze(0).expand(4, -1, -1)[:, 0::2].to(betas.device).float(), betas), 2). Now it report error return torch._VF.split_with_sizes(self, split_size, dim) RuntimeError: split_with_sizes expects split_sizes to sum exactly to 175 (input tensor's size at dimension 1), but got split_sizes=[1, 3, 3, 63, 10, 12, 12, 3, 3, 3, 10]
3. When I run the test_viz.py, it report error FileNotFoundError: [Errno 2] No such file or directory: 'data/cam_36.json'

[olivia23333]

1. The first warning comes from the self.register_buffer('xxx', xxx, persistent=False) part, which means don't save parameters. So these parameters will not be saved to the last checkpoint and will be reloaded when running inference, which should not affect the final result.
2. This error occurs because the original test.py uses the SMPLX parameters from the THuman dataset, which uses 12 dimensions to represent gestures, while the SMPLX parameter provided by the AMASS dataset uses 45 dimensions to represent gestures, so the size is incorrect. You need to change split_sizes=[1, 3, 3, 63, 10, 12, 12, 3, 3, 3, 10] to split_sizes=[1, 3, 3, 63, 10, 45, 45, 3, 3, 3, 10].
3. I will upload cam_36.json tomorrow because the server containing it is under repair and will be back tomorrow.

For inference code, I only tested the test.py, so the other inference codes contain errors, sorry for that. I will update them in this week.

[fluide1022]

thank u for reply

[fluide1022]

and also [1, 3, 3, 63, 10, 45, 45, 3, 3, 3, 10] sum is not equal to 175, still the above error

[olivia23333]

Sorry for the delay, animation code is updated now, and you can pull the latest code.

[fluide1022]

Great! I can run! But the results don't support facial expression animation.

https://github.com/user-attachments/assets/a2ef7375-3cfb-4de7-9475-2469cda4dd62

[olivia23333]

I am testing the code with AMASS CMU, which includes common expressions. Therefore, I do not add expression parameters in the code for simplification. To enable expression control, you can modify the animate_3d function in E3Gen/lib/apis/test.py. Append expression parameters to the smplx_param = np.concatenate section and update the prepare_deformer function in E3Gen/lib/models/deformers/smplx_deformer.py. For the check if smpl_params.shape[1] == 175: # smpl_params from amass, adjust the shape size and use torch.split(smpl_params, []) to access the expression parameters. Replace expression = torch.zeros(num_scenes, 10).to(device).float() with this modification. Following these steps, you will gain control over facial expressions.