search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
9.53k stars 1.3k forks source link

Splatfacto: use z-depth for supervision. #3345

Open michael-vedeler opened 3 months ago

michael-vedeler commented 3 months ago

I am working on using gaussian splat models with data directly from 3D software - such as blender, and there i am able to generate my own .ply pointclouds and all necessary passes in order to start training. The idea here is to be able to use gsplat as an alternative way to render out 3d models and display them easily on web.

I have managed to get really great results using my own pointclouds together with image-renders and masks. So the only thing missing now to get rid of any 'noisy' splats is to use the z-depth pass as training data as well.

(little demo of the results so far) https://github.com/user-attachments/assets/5f18e4d0-7491-40b9-8157-8b58c53ce670

michael-vedeler commented 3 months ago

I will ofc help out with delivering the files needed in order to test with.

maturk commented 3 months ago

@michael-vedeler this should be very much doable. You can check out my project dn-splatter where I add depth (z-depth or euclidean depth) supervision as well as some other things to the base splatfacto method. Repo here