Light estimation: Nvdiffrecmc vs nvdiffrec

[Selozhd]

I am concerned about the differences in light estimation for nvdiffrecmc compared to nvdiffrec. It appears at first sight that the light estimation takes some qualities of the material.

I am following the same procedure and config, which I described in a similar issue I opened in nvdiffrec for light estimation. (except reduced dmtet_grid to 128 from 256 due to gpu memory issues, and the new defualt 3000.0 for the laplace_scale) Shortly, I am generating a synthetic dataset in blender under a given environment map and give the model the same dataset for joint light, material, and mesh optimization.

Here are the environment map I gave, and light estimations by nvdiffrecmc and nvdiffrec respectively:

• original light
• probe.hdr estimated by nvdiffrecmc
• probe.hdr estimated by nvdiffrec

As you can see the light estimation has a blue hue. Furthermore, I think the light estimation also affects the quality of the final mesh. I am sharing a photograph below from the final mesh and an nvdiffrecmc rendering from the training. As you can see, the front of the shoe is missing in the mesh while the nvdiffrecmc render looks ok. I never experienced this kind of mesh deformation in nvdiffrec.

[JHnvidia]

Hello,

In the original nvdiffrec we used a simple white-balance regularizer for the light (code is available here: https://github.com/NVlabs/nvdiffrec/blob/main/render/light.py#L81). We removed it at some point while writing the new paper, but didn't have really strong arguments either way. It should be relatively easy for you to add back in.

There's no real right or wrong here, since it's impossible to disambiguate a blue materal from a scene with blue lighting. For photo-capture it's probably a good idea to have some form of regularizer that mimics natural sunlight.

[Selozhd]

Oh thanks, I didn't realize this! Will try and get back to you with the results.

[Selozhd]

I have rerun the trial with the same config, adding the white-balance regularization. Here is the new probe.hdr and a photograph of the final mesh. Just looking at these images, I think there is a considerable improvement.

However, I still don't understand why the sole has a hole.

[JHnvidia]

We've changed geometry regularization a bit to favor more "smooth" surfaces, by applying a geometric normal regularizer. https://github.com/NVlabs/nvdiffrecmc/blob/main/render/regularizer.py#L48 This helped some scenes, particularly the ones with specular reflections, but degraded quality in scenes with complex geometry (e.g. NeRF-Lego).

More importantly, though, geometric optimization is considerably more complex when including the shadowing term. Adding or carving out geometry can considerably change the shading of a part of the scene, which causes discontinuities in the training. It could be the cause of the hole, but I've mostly noted the system failing or being to conservative in carving out geometry. We're adding the shadow term incrementally during training, so you could try to delay the scheduling, or defer lighting/shadowing entirely to the system's second pass. https://github.com/NVlabs/nvdiffrecmc/blob/main/geometry/dmtet.py#L220

It should also be noted that we see the most benefit of the new system on scenes with strong directional lighting / complex shadowing (which is still hard). In your case, with a very smooth ambient light and limited shadowing (except for the inside of the shoe) it might be actually better to take some of the regularization ideas from the new paper and use the old system.

[Selozhd]

I implemented the shadowing delay used a more complex lighting. It appears that the shadowing delay is very helpful, delaying it to the second pass gave me the best results and removed the hole completely. This is also in good contrast to nvdiffrec where the sole was filled and in a dark colour.

@JHnvidia, thank you for your comments!