ShenhanQian
commented
8 hours ago Hi, thanks for sharing the detailed analysis. The problem is with a scaling factor and the global orientation of the world space.
-
Since you do not calibrate with a checkerboard but directly run SfM on multi-view images, the camera translations and point cloud are up-to-scaling, meaning the program can rescale these values together with an arbitrary scaling factor without breaking the geometric consistency. So, I was wrong about suggesting 100x or 1000x scaling because the unit of SfM does not correspond to any physical units such as mm or cm. What you can do is to optimize for this scaling factor during the rigid alignment stage with landmark. Basically, you initialize the scaling factor to 1 or exp(0), then use it to scale camera translations before rasterizating landmarks. Then, the landmark loss will help you to get the right scale.
-
In a similar spirit to the up-to-scale problem, the placement of cameras is also arbitrary, meaning the program can rotate and translate the cameras and point cloud as a whole to an arbitrary space without breaking multiview consistency. This leads to the difference in your Fig. 8. The solution is simple. In the coordinate space of FLAME, assuming we have an OpenGL camera always looking at FLAME (origin), what orientation should it be if we want this camera to view from the front of FLAME? It should be "x-right, y-up, z-backward". If not, we should apply a global rotation to the camera to fix this. For multiple cameras, we can do the same thing to the averaged orientations, which is exactly what we do in this function. Note that, if you have not converted your camera to OpenGL convention, the target orientation should change according to your current convention. That's why this function takes in 'target-convention' as an argument.
Jp-17
Background: Continuing the discussion from the previous issue（https://github.com/ShenhanQian/VHAP/issues/7#issuecomment-2402498210）, we have collected our own multi-view human data with 16 camera angles. We are currently using RealityCapture to estimate camera parameters and intend to use these parameters for VHAP calibration.
Problem encountered: The camera parameters obtained from RealityCapture cannot be directly used in the VHAP calibration process.
Exploration of causes:
First, using RealityCapture for camera parameter estimation, the overall camera array layout estimation appears relatively accurate (as shown in Figures 1 and 2, which are reconstructed human figures and camera arrangements using single-frame images from 16 viewpoints of nersemble_306 & our own captured human subject). Therefore, I estimate that the camera parameters obtained from RealityCapture should theoretically be usable.
However, when directly using the camera parameters obtained from RealityCapture for VHAP calibration, the FLAME landmarks converge together (as shown in Figure 3, red points). Adjusting the camera extrinsic translation to 1/100 or 1/200 of the original partially alleviates this phenomenon (Figure 4) , but still doesn't match the ground truth, with the scale being either too small or too large respectively.
I further examined the point cloud data generated by RealityCapture during the reconstruction of nersemble_306 (Figure 5) and compared it with the point cloud data of nersemble_306 previously obtained from gaussian_avatars training (Figure 6). I found that the xyz values of the two are clearly not on the same scale, with the former's values being significantly larger by over a hundred times (the point cloud data of nersemble_306 obtained from gaussian_avatars training is similar to the xyz data of FLAME points, both between -1 and 1).
Apart from the size issue, when plotting the point clouds from step 3, it can be seen that the orientations of the two 306 point clouds in the world coordinate system are also different (Figure 7).
To further observe the phenomenon in 4, I plotted the FLAME head points and the camera arrays of both (Figure 8, where the translation of the camera parameters obtained from RealityCapture on the right has been divided by 200). Indeed, although the FLAME head is the same, the world coordinates of the camera array estimated by RealityCapture are not coupled with the FLAME head (see Figure 9, where the FLAME head's face is facing the positive z-axis direction, with the head in the positive y-axis direction), while the facial orientation estimated by RealityCapture is not coupled with the z-axis.
Current speculations and areas of confusion:
a. Summarizing the above observations, the scale and world coordinate system (origin and xyz axis directions) of the camera parameters obtained from RealityCapture are inconsistent with the FLAME head. However, the camera parameters provided by the nersemble dataset can fully adapt to FLAME. How is this achieved in nersemble dataset?
b. In practice, I should be able to manually align the z-axis direction by adjusting the orientation of the human figure reconstructed by RealityCapture to face the front view directly, and modify the camera parameter translations by dividing by a scale factor. However, this method is inefficient. I'd like to hear your opinion on how the nersemble dataset construction and VHAP usage process achieve good alignment with the FLAME head.
Thank you!
Fig 1
Fig 2
Fig 3
Fig 4
Fig 5
Fig 6
Fig 7
Fig 8
Figure 9