The rendered images have some white blocks.

[silence401]

when I fix the mesh(without any material) to learn the material, the rendered images have some white blocks.

[jmunkberg]

Hello @silence401 ,

I need more details in order to help. In general, locking the geometry should only make things easier (I assume you learn materials and lighting jointly above). The config https://github.com/NVlabs/nvdiffrec/blob/main/configs/spot_metal.json shows one example of locking the geometry.

Here is an example of the bob model with locked geom as well, showing the starting point, and the result. I suspect some issue with your dataset above. Is the lighting constant in all captured frame and are the masks and poses accurate?

Config:

{
    "base_mesh"  : "data/bob/bob_tri.obj",
    "ref_mesh"  : "data/bob/bob_tri.obj",
    "random_textures": true,
    "iter": 1000,
    "save_interval": 100,
    "texture_res": [ 1024, 1024 ],
    "train_res": [512, 512],
    "batch": 4,
    "learning_rate": [0.03, 0.003],
    "ks_min" : [0, 0.25, 0],
    "envmap": "data/irrmaps/aerodynamics_workshop_2k.hdr",
    "env_scale" : 2.0,
    "background": "white",
    "validate" : false,
    "lock_pos" : true,
    "out_dir": "bob_fix_geo"
}

[silence401]

Thanks for your reply , I have a mesh without uv coodinate. So I run two pass, the first pass learn material['kd_ks_normal'] and then use xatlas to extract uv, finaly I run the second pass to learn material['ks'], material['kd'] etc. AND:

• the light is constant.
• the poses and masks is right

I Have Two Questions in fact:

1. the white blocks described above.(the start point also have)

2. the render result in first pass is better than second pass(which also have some noises ) First Pass: Second Pass:

[silence401]

Thanks for your reply , I have a mesh without uv coodinate. So I run two pass, the first pass learn material['kd_ks_normal'] and then use xatlas to extract uv, finaly I run the second pass to learn material['ks'], material['kd'] etc. AND:

• the light is constant.
• the poses and masks is right

I Have Two Questions in fact:

1. the white blocks described above.(the start point also have)
2. the render result in first pass is better than second pass(which also have some noises ) First Pass: Second Pass:

the white block is bug of nvdiffrast, I update it and error disapear.

[jmunkberg]

Nice that you got it working! Are you saying that you used an older version of nvdiffrast, and when updating to the latest version of nvdiffrast it works? Our installation instructions should pull the latest version of nvdiffrast automatically.

[silence401]

Nice that you got it working! Are you saying that you used an older version of nvdiffrast, and when updating to the latest version of nvdiffrast it works? Our installation instructions should pull the latest version of nvdiffrast automatically.

Yes, the nvdiffrast 0.2.7 not work but 0.2.8 work well, But I still have question about the second pass result is worse than the first pass descibed above.

[jmunkberg]

I still have question about the second pass result is worse than the first pass described above.

It is hard to say. In the second pass, we switch from volumetric texturing (using an MLP) to 2D textures, and, as discussed in the paper, in early stages of the second pass, texture seams are often visible, but usually disappear with some training. Some things to try:

• "texture_res": [ 2048, 2048 ], a too small texture res results in blurry textures in the second pass.
• "lock_pos" : true, Lock the geometry in the second pass and only optimize light and material
• "lock_light" : true, Lock the light in the second pass
• "learning_rate": [0.03, 0.003], try reducing/increasing the learning rate in the second pass (second value)