search

NVlabs / nvdiffrecmc

Official code for the NeurIPS 2022 paper "Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising".
Other
363 stars 28 forks source link

Initializing geometry with an existing mesh #2

Closed yyeboah closed 1 year ago

yyeboah commented 1 year ago

First off, cheers on the awesome work! I've managed to achieve some impressive material and light estimates by initializing with RGB images and it's become obvious that the geometry estimates could be further improved by initializing with an existing mesh, when available. Something akin to the training regime using the provided bob dataset.

I am attempting to do something similar and initialize the training using a textured mesh. The issue is, without throwing any errors, the estimates are all coming back blank, right from the beginning of training. I wonder if you've got any recommendations for how to properly initialize with meshes when BYOD'ing ?

jmunkberg commented 1 year ago

Thanks @yyeboah

See https://github.com/NVlabs/nvdiffrecmc/blob/main/configs/spot_metal.json for an example of a training with known geometry.

The flag "lock_pos" : true, disables position training, but you can instead leave it to false, and increase the Laplacian, to regularize the position training, play with these values (use relative with a large weight) 

    FLAGS.laplace             = "relative"  # Mesh Laplacian ["absolute", "relative"]
    FLAGS.laplace_scale       = 3000.0      # Weight for Laplace regularizer. Default is relative with large weight

To initialize the materials from an existing mesh, you need to hack a bit in the code I think. We do not support that with an existing flag, but should be straightforward to add.

yyeboah commented 1 year ago

Cheers @jmunkberg

The issue is resolved. Thanks for the prompt and insightful feedback.