Low Error - But Low Quality Tet Mesh

[9of9]

I've run a few tests with some in-the-wild examples from my photogrammetry collection, and I seem to be consistently running into the same issue: the combined output converges to something that is very, very similar to the target images, but the quality of actual tet mesh is garbage compared to what you get from the nerd data.

For instance, from 19 images:

The rendered output clearly converges pretty close to the desired result and kd seems maybe close-ish to what you'd want, but the probe, ks and n don't converge at all on what you would expect as correct values. The resulting tet mesh holds roughly the right shape, but generally just looks like swiss cheese:

Another example from 45 images:

This one generated a better mesh, but seemed to over-optimise ks for a surface which should largely be very diffuse. On the inside, the mesh is still very much swiss cheese.

This seems in contrast to very solid results on synthetic data. Do you have any recommendations for how to tune the training for better results? I have used similar configs to what are used for the nerd datasets, but I'm not sure if there are better ways of controlling the learning rates. Or is the trick ultimately that it just needs more photos?

Edit: Happy to share and open-source these datasets if that would be at all helpful for future testing.

[JHnvidia]

Hi @9of9,

I'm just speculating, but 19 views sounds like it's probably on the low side. Additionally you can expect mesh quality to degrade if camera poses and segmentation masks are inaccurate.

Since we're doing a full SDF grid for DMTet (as supposed to some smooth representation like an MLP) it is prone to fail towards a high frequency "mess". It's a tradeoff we had to make because overly smoothing geometry doesn't work that well with very complex scenes (e.g. Lego from the nerf dataset). It is possible to regularize geometry by using different reprensentations, and we did that for the DTU datasets in the paper (some info here: https://github.com/NVlabs/nvdiffrec/issues/9).

The specular term is hardest to train (and where most of our issues lie). If you know that your scenes only contain diffuse materials you can restrict the specular terms from the config. They are: [unused, roughness, metalness], so the following should force the optimizer to only consider rough diffuse-like surfaces.

    "ks_min" : [0, 1, 0],
    "ks_max" : [0, 1, 0],

You could also change to a diffuse BSDF by chaining the following line to = 'diffuse', but I havent tried that code path so I'm not 100% sure if it works. https://github.com/NVlabs/nvdiffrec/blob/main/train.py#L176

I would also suggest comparing the results of both passes, because sometimes the second pass can degrade geometric quality (but this doesn't seem to be the cause in your scenes). There's a "lock_pos" : true, flag to avoid moving the geometry in the second pass.

[9of9]

Thank you, that's very thorough! I will investigate and see if there's a way to improve the results :)

[tlok666]

Thank you, that's very thorough! I will investigate and see if there's a way to improve the results :)

Hi, I come across the same problem. Did you solve it by adjusting the ks values?

[9of9]

Thank you, that's very thorough! I will investigate and see if there's a way to improve the results :)

Hi, I come across the same problem. Did you solve it by adjusting the ks values?

It was better in some respects, but not dramatically. Here I constrained the lego house to be non-metallic and relatively shiny:

And same with the tower above that I constrained to be fairly rough and non-metallic:

The results are a little more sensible and conventional, it has made the diffuse in particular behave better, and the env maps generated from these are a little bit more plausible, but the quality of the tet meshes is still quite poor. I suspect the only real solution will be to use many more source images.

[tlok666]

Thank you, that's very thorough! I will investigate and see if there's a way to improve the results :)

Hi, I come across the same problem. Did you solve it by adjusting the ks values?

It was better in some respects, but not dramatically. Here I constrained the lego house to be non-metallic and relatively shiny:

And same with the tower above that I constrained to be fairly rough and non-metallic:

The results are a little more sensible and conventional, it has made the diffuse in particular behave better, and the env maps generated from these are a little bit more plausible, but the quality of the tet meshes is still quite poor. I suspect the only real solution will be to use many more source images.

It seems like, nvdiffrec could generate limited mesh quality in complicated cases. A related paper refers to the same issues of 'Low Error - But Low Quality Tet Mesh'. (https://arxiv.org/pdf/2206.03380.pdf)

[YLongJin]

Hello, bro I got the same issues as you(Low-Quality Tet Mesh)， but my result of img_dmtest is not better than yours. And tet mesh is like shit! But I still love it.

Could you plz tell me how to do that>?

1. What is your method of taking a photo?? say your pose, the resolution of your tool, and so on.,.
2. Could you plz tell me how to set the parameters of config file??

@JHnvidia

Best, Long