search

turandai / gaussian_surfels

Implementation of the SIGGRAPH 2024 conference paper "High-quality Surface Reconstruction using Gaussian Surfels".
540 stars 26 forks source link

Questions about calculating projection matrices #31

Closed CanCanZeng closed 5 months ago

CanCanZeng commented 5 months ago

Hi, I noticed that your CUDA code uses full_proj_transform https://github.com/turandai/gaussian_surfels/blob/c1cbb40b6f048e7a13535c8e0fb210c8a8192966/gaussian_renderer/__init__.py#L47

, which is derived from projection_matrix https://github.com/turandai/gaussian_surfels/blob/c1cbb40b6f048e7a13535c8e0fb210c8a8192966/scene/cameras.py#L97

but you does not take into account the principal point when calculating projection_matrix https://github.com/turandai/gaussian_surfels/blob/c1cbb40b6f048e7a13535c8e0fb210c8a8192966/scene/cameras.py#L72 https://github.com/turandai/gaussian_surfels/blob/c1cbb40b6f048e7a13535c8e0fb210c8a8192966/utils/graphics_utils.py#L66 So even though you used the principal point in the CUDA code, using the full_project_matrix may still result in imprecise results https://github.com/turandai/gaussian_surfels/blob/c1cbb40b6f048e7a13535c8e0fb210c8a8192966/submodules/diff-gaussian-rasterization/cuda_rasterizer/forward.cu#L254

Is what I am considering correct?

turandai commented 5 months ago

Hi, yes the projection matrix is imprecise as we consider the principal point here, but it seems better to do it in projection matrix.

CanCanZeng commented 5 months ago

Hi, I also noticed that you use principal point to recover point_image, my point is that the input p_proj is calculated from full_proj_transform, and full_proj_transform is not suitable for the imperfect pinhole camera, so the recovered point_image is still not accurate.

turandai commented 5 months ago

Hi, I think you are right. I didn't think deep here. I refered to this and updated the projection matrix. The previous modification in CUDA is now reversed. Thank you very much for your remider! I am not very sure if this is properly fixed now, please feel free to get in touch.

CanCanZeng commented 5 months ago

That's a good idea!