How to get a model with higer accuracy and more details?

[wuge1880]

Hello,

I have run "python train.py --config configs/nerf_lego.json" with default setting. The picture below shows my mesh output. 1) The smooth surface is not flat (as shown in the red box). How should I improve it? 2) Lots of details are loss due to the geometric simplification. How can I get a model with more details?

Thank you in advance!

[jmunkberg]

I think that is roughly the expected quality on that example. See Fig 30 in our paper: https://nvlabs.github.io/nvdiffrec/assets/paper.pdf Note that we store small scale details also in tangent space normal maps, so fine scale detail won't be visible in direct visualization of the mesh.

To improve, you could try playing with the the learning rates, the various regularizers and increase the batch size if you have big GPU or a multi-GPU setup. You could also disable normal maps from the optimization if you only care about the base mesh quality.

To increase the detail, you can start from a higher resolution tetrahedral grid. This is controlled by the config flag "dmtet_grid" : 128, . To keep the code repo small, we only included three variants: https://github.com/NVlabs/nvdiffrec/tree/main/data/tets, but please refer to the instructions here to generate higher resolution tet grids: https://github.com/NVlabs/nvdiffrec/blob/main/data/tets/generate_tets.py

[wuge1880]

Thank you for your reply. I have generated a tet grid with higher resolution (256), and run it with the following config. But the middle results seems to be wrong, as shown in the picture below. How should I fix it ?

Besides, I'd like to know how to disable normal maps from the optimization, is there a config flag to control it ?

{ "ref_mesh": "data/nerf_synthetic/lego", "random_textures": true, "iter": 5000, "save_interval": 100, "texture_res": [ 2048, 2048 ], "train_res": [800, 800], "batch": 10, "learning_rate": [0.03, 0.003], "ks_min" : [0, 0.25, 0.0], "dmtet_grid" : 256, "mesh_scale" : 2.4, "laplace_scale" : 3000, "display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}], "background" : "white", "out_dir": "nerf_lego" }

[jmunkberg]

In your reconstruction, it looks like the tet grid is not centered.

We assume that the tet grid is centered and in the range [-0.5, 0.5]. Add a print around line 174 in dmtet.py like print("tet grid min/max", torch.min(self.verts).item(), torch.max(self.verts).item()) . With a mesh_scale factor of 2.1, I get tet grid min/max -1.0499999523162842 1.0499999523162842 with our tet grids.

Below is the centered cube in wavefront OBJ we used to generate grids of varying resolutions in quartet

o cube_center
v -0.500000 0.500000 -0.500000
v -0.500000 0.500000 0.500000
v 0.500000 0.500000 0.500000
v -0.500000 -0.500000 -0.500000
v -0.500000 -0.500000 0.500000
v 0.500000 -0.500000 0.500000
v 0.500000 -0.500000 -0.500000
v 0.500000 0.500000 -0.500000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vt 0.000000 0.000000
vn 0.0000 1.0000 0.0000
vn -1.0000 0.0000 0.0000
vn 0.0000 -0.0000 1.0000
vn 1.0000 0.0000 0.0000
vn 0.0000 -1.0000 -0.0000
vn 0.0000 0.0000 -1.0000
s off
f 1/1/1 2/2/1 3/3/1
f 1/1/2 4/4/2 5/5/2
f 2/2/3 5/5/3 6/6/3
f 3/3/4 6/6/4 7/7/4
f 5/5/5 4/4/5 7/7/5
f 4/4/6 1/1/6 8/8/6
f 3/3/1 8/8/1 1/1/1
f 5/5/2 2/2/2 1/1/2
f 6/6/3 3/3/3 2/2/3
f 7/7/4 8/8/4 3/3/4
f 7/7/5 6/6/5 5/5/5
f 8/8/6 7/7/6 4/4/6

[jmunkberg]

To disable normal maps, in render.py around line 66, change from

if 'no_perturbed_nrm' in material and material['no_perturbed_nrm']:
   perturbed_nrm = None

to perturbed_nrm = None

[wuge1880]

Thank you! I'v used the centered cube ("dmtet_grid" : 256) and the middle results are back to normal. But the training process was stuck at xatlas.parametrize in train.py, line104 (for over an hour). If my major focus is the mesh quality, how to speed up this process?

[jmunkberg]

xatlas can be very slow or hang if the geometry reconstruction is bad, for example if the training diverges. It is hard to say, but if the first pass has done a good job on reconstructing the geometry, xatlas usually succeeds in creating a parameterization.

This is a pain point for our approach, but UV-parameterization is a very hard problem in the general case, and we rely on off-the-shelf tools for that. If you only care about the geometry and don't need textures, you could bypass uvatlas and the second optimization step (need to hack the code, not exposed in the config).

[wuge1880]

I will try it. Thank you for the detailed reply!