Open fluide1022 opened 1 month ago
You can try changing the key "cond_poses" to "test_poses", the rendered result needs to be flipped horizontally and vertically to get the final animation results. If it still doesn't work, please update the error below, I will check it again. Sorry for the inconvenience.
unexpected key in source state_dict: decoder.init_pcd, decoder.init_rot, decoder.face_mask, decoder.hands_mask, decoder.outside_mask, decoder.deformer.init_spdir, decoder.deformer.init_podir, decoder.deformer.init_faces, decoder.deformer.init_lbsw, decoder.deformer.tfs_inv_t, decoder.deformer.vs_template, decoder.deformer.smpl_faces, decoder.deformer.pose_offset_cano, decoder_ema.init_pcd, decoder_ema.init_rot, decoder_ema.face_mask, decoder_ema.hands_mask, decoder_ema.outside_mask, decoder_ema.deformer.init_spdir, decoder_ema.deformer.init_podir, decoder_ema.deformer.init_faces, decoder_ema.deformer.init_lbsw, decoder_ema.deformer.tfs_inv_t, decoder_ema.deformer.vs_template, decoder_ema.deformer.smpl_faces, decoder_ema.deformer.pose_offset_cano
.
but it does not influence the code running.data['test_poses'] = data['test_poses'][:, 31:32] data['test_intrinsics'] = data['test_intrinsics'][:, 31:32] betas = data['test_smpl_param'][:, 70:80].unsqueeze(1).expand(-1, 50, -1).float() data['test_smpl_param'] = torch.cat((smplx_param.unsqueeze(0).expand(4, -1, -1)[:, 0::2].to(betas.device).float(), betas), 2)
.
Now it report error return torch._VF.split_with_sizes(self, split_size, dim) RuntimeError: split_with_sizes expects split_sizes to sum exactly to 175 (input tensor's size at dimension 1), but got split_sizes=[1, 3, 3, 63, 10, 12, 12, 3, 3, 3, 10]
FileNotFoundError: [Errno 2] No such file or directory: 'data/cam_36.json'
self.register_buffer('xxx', xxx, persistent=False)
part, which means don't save parameters. So these parameters will not be saved to the last checkpoint and will be reloaded when running inference, which should not affect the final result.test.py
uses the SMPLX parameters from the THuman dataset, which uses 12 dimensions to represent gestures, while the SMPLX parameter provided by the AMASS dataset uses 45 dimensions to represent gestures, so the size is incorrect. You need to change split_sizes=[1, 3, 3, 63, 10, 12, 12, 3, 3, 3, 10]
to split_sizes=[1, 3, 3, 63, 10, 45, 45, 3, 3, 3, 10]
.For inference code, I only tested the test.py
, so the other inference codes contain errors, sorry for that. I will update them in this week.
thank u for reply
and also [1, 3, 3, 63, 10, 45, 45, 3, 3, 3, 10] sum is not equal to 175, still the above error
Sorry for the delay, animation code is updated now, and you can pull the latest code.
Great! I can run! But the results don't support facial expression animation.
https://github.com/user-attachments/assets/a2ef7375-3cfb-4de7-9475-2469cda4dd62
I am testing the code with AMASS CMU, which includes common expressions. Therefore, I do not add expression parameters in the code for simplification. To enable expression control, you can modify the animate_3d
function in E3Gen/lib/apis/test.py
. Append expression parameters to the smplx_param = np.concatenate
section and update the prepare_deformer
function in E3Gen/lib/models/deformers/smplx_deformer.py
. For the check if smpl_params.shape[1] == 175: # smpl_params from amass
, adjust the shape size and use torch.split(smpl_params, [])
to access the expression parameters. Replace expression = torch.zeros(num_scenes, 10).to(device).float()
with this modification. Following these steps, you will gain control over facial expressions.
When running the novel pose animation:
python demo/test_ani.py configs/ssdnerf_avatar_uncond_thuman_conv_16bit.py work_dirs/ssdnerf_avatar_uncond_16bit_thuman_conv/ckpt/ssdnerf_avatar_uncond_16bit_thuman_conv/latest.pth --gpu-ids 0
It report error:~/E3Gen/lib/apis/test.py", line 238, in animate_3d data['cond_poses'] = data['cond_poses'][:, 31:32] KeyError: 'cond_poses'