search

zju3dv / EasyMocap

Make human motion capture easier.
Other
3.63k stars 449 forks source link

Instance and Label Masks for Novel View Synthesis Work (SIGGRAPH 2022) #255

Open metro-merl opened 1 year ago

metro-merl commented 1 year ago

Hi @chingswy, @cvxzhou, and others,

I had a query about the pre-processed soccer data provided as part of the code release of the paper, "Novel View Synthesis of Human Interactions from Sparse Multi-view Videos". In it, the output-smpl-3d folder comes with instance-level masks of the humans in the scene. Can someone please clarify how these are extracted? Also, how are occlusions handled?

Besides, how are the stick figures in the labels folder under output-smpl-3d obtained? Are they based on the learnt SMPL parameters or is it just a 2d pose estimate?

Also on a related note, how is the mask for the ground plane computed?

Appreciate any help!

chingswy commented 1 year ago

Hello, the instance-level masks are extracted from SMPL estimation, as well as the labels.

metro-merl commented 1 year ago

@chingswy Really appreciate that information. Would you mind pointing me to which code block does this? I was following the steps in this repo to get the SMPL model. https://github.com/zju3dv/EasyMocap/blob/master/doc/mvmp.md

Also, how are the ground plane masks obtained?

sycamore27 commented 1 year ago

@chingswy Hello, thanks for the great work! I have a similar question. How do you get the labels data in MultiNeuroalBody? Is it gotten by manual annotation or other detection method? Look forward to your reply, thank you!

chingswy commented 1 year ago

Hello, the label for the soccer is gotten by YOLO. We have trained a new yolo with few annotated images.

chingswy commented 1 year ago

@chingswy Really appreciate that information. Would you mind pointing me to which code block does this? I was following the steps in this repo to get the SMPL model. https://github.com/zju3dv/EasyMocap/blob/master/doc/mvmp.md

Also, how are the ground plane masks obtained?

This part is not available now, I will add it in later version. We don't need the ground plane masks. The provided masks are the region of air, which are roughly annotated.

metro-merl commented 1 year ago

Hi @chingswy, Do you have an estimate of when this part of the code (SMPL -> Instance Masks and Labels) would become available?

Feez2403 commented 5 months ago

@chingswy I'm also interesting in the way to get the "label" masks from the 3d keypoints / smpl parameters