search

SimonGiebenhain / MonoNPHM

[CVPR 2024 Highlight]
https://simongiebenhain.github.io/MonoNPHM/
Other
112 stars 20 forks source link

Inputs and outputs, canonical space #6

Open Zvyozdo4ka opened 4 months ago

Zvyozdo4ka commented 4 months ago

Great Project, thank you for sharing

  1. May i ask you, what is the input and output data in this project? Would be the output as a mesh?
  2. and what is the definition of canonical space, geometry, coordinates?
  3. Would be the coordinate system same as in NPHM?
SimonGiebenhain commented 4 months ago

Hi, thanks for your interest.

  1. The input can be a single image, or a video. The output would be the latent codes (geometry, appearance and expression) for the person in the image, or for each frame of the video. From the latent codes which describe a Signed Distance Field, we can reconstruct a mesh using marching cubes. So the mesh could also be considered as the output.
  2. In the canonical space, we define the signed distance field (SDF) describing the geometry of a person. The canonical space has a sort of neutral facial expression. In the same fashion, we define a texture field in the canonical space, which describes the appearance/color of a person. The facial expressions are described using a deformation field, which predicts the canonical coordinates for any 3D point of an "actual" face with some facial expression. So in some sense, the deformation field removes any influence of the facial expression.
  3. The coordinate system is identical to the of NPHM.

Let me know if you have more questions.

Zvyozdo4ka commented 4 months ago

Thank you for your quick and detailed response!

May i ask you if you compared the quality of output mesh of NPHM and MonoNPHM? And what influence the quality of mesh the most? I noticed that for NPHM input point cloud quality affects output mesh very much. To set up environment is so tricky, i can't compare by myself.

SimonGiebenhain commented 4 months ago

I think MonoNPHM behaves more robust in general. Since there are many technical differences, also in the training procedure, it is hard to say what exactly helps how much.

What problems do you have during the set up?

Zvyozdo4ka commented 4 months ago

What problems do you have during the set up?

since my cuda_11.5, i could not install torchaudio properly, and torchvision is 0.18.1, pytorch is 1.12.0.

Then when i ran conda install pytorch3d=0.7.4 -c pytorch3d took several hours, then finally was installed.

Megidd commented 3 months ago

@Zvyozdo4ka I followed the exact same instructions on the README section of Installation Dependencies. I ran into lots of HTTP errors, but repeating the instructions again and again, they finally got installed.