question about refined_ZJU_MoCap

[yejr0229]

Hi,I'd like to know the detailed description of refined_ZJU_MoCap,especially about the "colmap" ,"params","annots.npy","schp" and the "smpl_meta" data,thank you so much.

[yejr0229]

And I also find instant-nvr can train with normal map,but it seems like the network can't predict normal map after I check the code,can you tell me how to train with normal map?

[gcgeng]

Basically annots.npy stores the camera pose parameter, schp is the human instance mask, smpl_meta is some meta-data for smpl parametric model. Please refer to the code for details. The current method does not support normal supervision. I think you need to write/modify some new code to support this feature. Please feel free to reopen the issue if you have further questions.

[yejr0229]

Could you please tell me roughly how to modify the code? Thank you so much.

[gcgeng]

Could you please tell me roughly how to modify the code? Thank you so much.

It is not straightforward to modify the code to support normal supervision. Ideally, you should first change the underlying geometry representation to SDF and compute the normal either by autograd or numerical computing. Afterward, you should use some off-the-shelf normal estimation models like ICON or ECON to get a predicted normal map. Finally, you can compute the difference between the rendered normal map and the predicted normal map which can be used as an additional supervision.

[yejr0229]

Thanks for replying.I follow your advice to change the underlying geometry representation to SDF,but I found 'my_xxx/lbs/' folder in dataset didn't contain the ground truth of SDF:'bigpose_sdf.npy'.I found the following code to compute it,but I'm having problem install OpenGl so the mesh_to_sdf package can't work: mesh = trimesh.Trimesh(bigpose_vertices, faces) points, sdf = mesh_to_sdf.sample_sdf_near_surface(mesh, number_of_points=250000) translation, scale = compute_unit_sphere_transform(mesh) points = (points / scale) - translation sdf /= scale sdf_path = os.path.join(lbs_root, 'bigpose_sdf.npy') np.save(sdf_path, {'points': points, 'sdf': sdf}) I wonder could you send me the 'bigpose_sdf.npy' for each actor,thank you so much!

[gcgeng]

The bigpose_sdf.npy is not used in our method. If you are referring to the code in https://github.com/zju3dv/instant-nvr/blob/83d369711a77a019460bd0c9622126a8195a7d9a/tools/prepare_zjumocap.py#L397 this snippet is commented out and is deprecated. Moreover, as a clarification, bigpose_sdf.npy is computed using SMPL's mesh, which should NOT be used as your SDF or normal supervision.

[yejr0229]

Thanks for replying.And I wonder how did you get the uv coordinates for smpl body,denote as 'smpl_uv.obj' in your code.

[yejr0229]

Also you find 4 cloest smpl_vertices in each body part for every querypoint,and compute average distance for querypoint to 5 body parts,then you set smpl_thresh=0.05 to choose the body part for querypoint,I found it will cause more than one body part assign to one point,and part that too far will not be computed in hash encoding,I don't know whether I'm right above all... I'm a little confused about this,I think one point suppose only assign to one body part otherwise information in different part will mixed,for example,if a point is assigned to the hand and head simultaneously,then it will contain both head and hand information,during rendering the rgb will mixed and caused bad affect.

[gcgeng]

Thanks for replying.And I wonder how did you get the uv coordinates for smpl body,denote as 'smpl_uv.obj' in your code.

If I recall correctly, the uv map is provided in the SMPL official release.

[gcgeng]

Also you find 4 cloest smpl_vertices in each body part for every querypoint,and compute average distance for querypoint to 5 body parts,then you set smpl_thresh=0.05 to choose the body part for querypoint,I found it will cause more than one body part assign to one point,and part that too far will not be computed in hash encoding,I don't know whether I'm right above all... I'm a little confused about this,I think one point suppose only assign to one body part otherwise information in different part will mixed,for example,if a point is assigned to the hand and head simultaneously,then it will contain both head and hand information,during rendering the rgb will mixed and caused bad affect.

We have a multi-part aggregation mechanism to resolve this issue. Please refer to Eq. 8 in our paper.

[alvaro-budria]

Thanks for replying.I follow your advice to change the underlying geometry representation to SDF,but I found 'my_xxx/lbs/' folder in dataset didn't contain the ground truth of SDF:'bigpose_sdf.npy'.I found the following code to compute it,but I'm having problem install OpenGl so the mesh_to_sdf package can't work: mesh = trimesh.Trimesh(bigpose_vertices, faces) points, sdf = mesh_to_sdf.sample_sdf_near_surface(mesh, number_of_points=250000) translation, scale = compute_unit_sphere_transform(mesh) points = (points / scale) - translation sdf /= scale sdf_path = os.path.join(lbs_root, 'bigpose_sdf.npy') np.save(sdf_path, {'points': points, 'sdf': sdf}) I wonder could you send me the 'bigpose_sdf.npy' for each actor,thank you so much!

Hi @yejr0229, did you manage to obtain high quality renderings with an SDF-based representation? Do you mind sharing the changes you did in the code?