search

autonomousvision / differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf
MIT License
794 stars 90 forks source link

Is it possible to have a scene-general model for multi-view reconstruction? #62

Closed Na-Z closed 2 years ago

Na-Z commented 3 years ago

Hi. Thanks for sharing your awesome work.

I noticed that for the multi-view reconstruction, your method requires to re-train the model for each scene, i.e., scene-specific. I am curious to know if it is possible to make the model scene-general, which means it can reconstruct the scene with multi-view inputs.

Thanks.

m-niemeyer commented 2 years ago

Hi @Na-Z , thanks for your interest in the project!

This is definitely possible, but we haven't explored this in this work. The two most common ways I would see to approach this is to a.) design a multi-view encoder, or b.) perform test-time optimization with multiple views. This, in fact, you could also do with our code where you would basically combine both approaches; first, use the single-view model to obtain a good estimate, and then fine-tune this by optimizing for the given multi-view images.