search

monniert / differentiable-blocksworld

[NeurIPS 2023] Code for "Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives"
https://www.tmonnier.com/DBW
MIT License
197 stars 10 forks source link

Question about the design of the model #5

Closed crepejung00 closed 1 year ago

crepejung00 commented 1 year ago

Hi, thanks for the awesome work! After reading the paper and codes I have one question about the model design. In section 3, you mention as

Note that compared to recent advances in neural volumetric representations [50, 45, 76], we do not use any neural network and directly optimize meshes, which are straightforward to use in computer graphic pipelines.

I was curious about the part about not using any neural networks because as the model is already light to train, I thought that attaching a small MLP layer after the primitive can highly increase the quality of the rendered images as from other NeRF type models. Although I do understand that attaching the MLP will make the entire model less practical in the sense that we cannot use it in computer graphics pipelines, I think this will make the model competitive with other existing MVS models. Have you ever conducted any experiments with neural networks?

Thanks!

monniert commented 1 year ago

Hi @crepejung00 thanks for reaching out!

That's a good point, we discussed it as some point to also increase the training speed but did not investigate it further. In general, I think it would be highly beneficial to model textures with MLPs because it will allow to automatically allocate memory to texture regions that require a high-level of details, which is an issue in our current formulation.