color noise on texture if use base_mesh

[AY1997]

hi, thanks for your impressive work! I use lock_pos for bob dataset like json file you mentioned in https://github.com/NVlabs/nvdiffrec/issues/22

{ "base_mesh" : "data/bob/bob_tri.obj", "ref_mesh" : "data/bob/bob_tri.obj", "random_textures": true, "iter": 1000, "save_interval": 100, "texture_res": [ 1024, 1024 ], "train_res": [512, 512], "batch": 4, "learning_rate": [0.03, 0.003], "ks_min" : [0, 0.25, 0], "envmap": "data/irrmaps/aerodynamics_workshop_2k.hdr", "env_scale" : 2.0, "background": "white", "validate" : false, "lock_pos" : true, "out_dir": "bob_fix_geo" }

then I get the kd texture results like:

There will be a lot of rgb noise on the texture map, and similar issue happens for other datasets as long as I use the base_mesh. why is this happened？And do you have any possible suggestions？

[jmunkberg]

Thanks!

We train all mip levels of the textures, so if some high levels mips are not used in the rendering, there may be noise left there. I assume that the rendered images dumped after some training do not show noisy textures?

To circumvent this, either increase the training resolution, say "train_res": [1024, 1024], or, to verify that this is indeed the cause of the noise, keep the rendering resolution at 512, but decrease the texture resolution drastically, say to "texture_res": [ 128, 128 ],, and check that the noise is gone.

A rule-of-thumb is to keep the rendering and texture resolution roughly equal.

[AY1997]

Thanks!

We train all mip levels of the textures, so if some high levels mips are not used in the rendering, there may be noise left there. I assume that the rendered images dumped after some training do not show noisy textures?

To circumvent this, either increase the training resolution, say "train_res": [1024, 1024], or, to verify that this is indeed the cause of the noise, keep the rendering resolution at 512, but decrease the texture resolution drastically, say to "texture_res": [ 128, 128 ],, and check that the noise is gone.

A rule-of-thumb is to keep the rendering and texture resolution roughly equal.

At the right camera pose, the rendered image looks normal. I kept the other configs and decrease the texture resolution to [128, 128], and still get an incorrect kd texture:

And the noise still exists. I guess this could be caused by incorrect initialization or precision errors of the colors, or something else. Do you have any other suggestions ? I also meet this problem when I use base_mesh while ref_mesh is an image dataset with corresponding camera pose.

[jmunkberg]

AFAICT from your latest texture, there is no longer any noise in the part of the UV space that is covered, right? We do initialize the textures as noise, and in regions not queried, we won't get any gradients, so those texels will not get any updates so noisy regions outside the object in UV space is expected.

For reference, here are the UVs visualized in Blender

[AY1997]

AFAICT from your latest texture, there is no longer any noise in the part of the UV space that is covered, right? We do initialize the textures as noise, and in regions not queried, we won't get any gradients, so those texels will not get any updates so noisy regions outside the object in UV space is expected.

For reference, here are the UVs visualized in Blender

yes I get your point ! I tried lower texture resolution and higher train resolution and it works, noise is much less. There is still a few noise at the border of the queried region, which I guess is not fully converged. Thanks for your easonable explanation！