search

autonomousvision / differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf
MIT License
798 stars 90 forks source link

Pre-train / initialization #12

Closed esx2ve closed 4 years ago

esx2ve commented 4 years ago

Hi! Amazing paper, and very clean code! Thanks so much for releasing this.

I have a somewhat practical question. Is it possible to somehow "pre-train" the network, or initialize it so the optimization process just "fine-tunes" the little details? I have the option of having a pretty high-quality triangle mesh output from a different algorithm. Is it possible in principle to come up with a scheme that will allow this?

The motivation is mainly reducing training time, and I can sample the SDF easily from my existing reconstruction output.

Thanks again!

m-niemeyer commented 4 years ago

Hi @esx2ve , thanks so much for you interest and kind words!

Regarding your question, I agree that it is an interesting problem! However, one downside of these neural representations is that shape and texture is basically stored in the weights of the network, so it is not so trivial to manipulate this. What could speed up training in theory is that if you have access to some 3D supervision (as you describe it), you could first train the network with this - for example with uniformly sampled points in 3D space and their occupancy value. You would then apply a binary cross entropy loss to the predicted occupancy and the true occupancy - just how we did it in our previous project Occupancy Networks. You could later switch to the depth map-based or RGB-based losses to fine-tune / learn the texture. Good luck with your research!