ziyc
commented
2 months ago Hi @sqchai, thanks for your interest!
Unfortunately, due to the whole image rendering scheme of 3DGS, we can't un-distort images as in NeRF-based (ray-based) methods. To the best of our knowledge (please correct me if I've missed any works), there's no 3DGS that incorporates distortion parameters for rasterization, except for 3D Gaussian Ray Tracing, which adapts Gaussian rendering to ray-based rendering.
In our codebase, we use the cv2.undistort function provided by OpenCV. This function undistorts images by correcting pixel positions and cropping the field of view. However, this undistortion method has limitations; it can't handle severely distorted images, as it significantly reduces the field of view. I found an example here
We tested this undistortion method on Nuplan (which has severe distortion issues), but it didn't work well, so we turned it off. Based on our past experiments, this undistortion approach might not be suitable for all datasets. However, in Waymo, where the distortion isn't as severe, we can use it. As for other datasets, we either couldn't find distortion parameters in the dataset, or we found this distortion method doesn't work, so we turned it off.
In conclusion, this is not an ideal undistortion method, and we maintain a conservative attitude towards its use.
I think image distortion in 3DGS needs further investigation, welcome discussion!
Thanks for your great work!
Why is the "undistort" parameter set to "True" only for the Waymo dataset? I think images from other datasets (Panda, Nuplan, ...) also exhibit distortion that requires handling?
Thanks in advance!