search

apple / ml-neuman

Official repository of NeuMan: Neural Human Radiance Field from a Single Video (ECCV 2022)
Other
1.26k stars 141 forks source link

Question about datasat? #99

Open Qmebius opened 3 months ago

Qmebius commented 3 months ago

I would like to ask if the dataset provides the 2D keypoints of SMPL. I have downloade but not found

jiangwei221 commented 3 months ago

We used mmpose to detect the 2D keypoints: https://github.com/apple/ml-neuman/blob/0149d258b2afe6ef65c91557bba9f874675871e4/preprocess/gen_run.py#L95-L104

Qmebius commented 3 months ago

Thanks，I would also like to ask, the bin files provided in the dataset contain depth information, which is very sparse in the human body area. Is there any way to obtain the depth information of the human body?

jiangwei221 commented 3 months ago

Yes, we run BoostingMonocularDepth to estimate the human depth and fuse with colmap MVS depth.

  1. BoostingMonocularDepth: https://github.com/apple/ml-neuman/blob/0149d258b2afe6ef65c91557bba9f874675871e4/preprocess/gen_run.py#L106-L121
  2. Depth fusion: https://github.com/apple/ml-neuman/blob/0149d258b2afe6ef65c91557bba9f874675871e4/data_io/neuman_helper.py#L77-L86
Qmebius commented 3 months ago

And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position?

jiangwei221 commented 2 months ago

And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position?

We don't have a "ground truth camera pose" which usually requires manual annotations or extra hardwares. We use COLMAP to reconstruct the "reference camera poses".

Qmebius commented 2 months ago

So can this reference pose be used as the ground truth for the camera tracking task？

---Original--- From: "Wei @.> Date: Wed, Jul 10, 2024 01:00 AM To: @.>; Cc: @.**@.>; Subject: Re: [apple/ml-neuman] Question about datasat? (Issue #99)

And I also want to ask how you obtain the ground truth of the camera pose, and is the 'trans' processed by alignment.py the gt value of the human body's position?

We don't have a "ground truth camera pose" which usually requires manual annotations or extra hardwares. We use COLMAP to reconstruct the "reference camera poses".

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>