search

dcharatan / pixelsplat

[CVPR 2024 Oral, Best Paper Runner-Up] Code for "pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction" by David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann
http://davidcharatan.com/pixelsplat/
MIT License
864 stars 60 forks source link

How to add depth loss #66

Closed SongHaixu closed 5 months ago

SongHaixu commented 6 months ago

Thank you for your excellent work! I want to increase depth loss for my own dataset during the training process. However, I notice that the depth map rendered by Gaussian kernel is colored rather than grayscale. Therefore, when I have the scaled depth value of each pixel's in the image, what should I do to leverage gt depth values to supervise the depth map rendered by Gaussian?

dcharatan commented 5 months ago

You should be able to use this function, which will give you the depth without the color map applied. However, because depth supervision/regularization wasn't a central part of the paper, this approach to rendering depth is kind of hacky (we use the Gaussians' depths as their colors, then render these colors). It might be worth checking out forks of diff-gaussian-rasterization that natively support depth rendering, e.g. this one. Disclaimer: I haven't actually tried the linked repo out myself.