search

NVlabs / nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Other
2.09k stars 222 forks source link

Normal map for training #157

Open Kev1MSL opened 2 months ago

Kev1MSL commented 2 months ago

Hi! I am a bit confused in the training part using meshes. I am trying to use the normal map as a parameter for supervision for guiding the loss function. My goal would be to build a dataset from .obj files and use the renderer for creating the images, as well as their normal map. You mentioned in #85 that normal map was directly derived from the .obj file for training, but I am wondering how the supervision could work if the normal map derived from the renderer is smoothed but the one from the .obj file is fine. If I am rendering the normal map of a cube I am getting something like that: normal_map When I would normally expect the normal map render to look like that: cube_normal

Not sure if I am missing something...

jmunkberg commented 2 months ago

Hello @Kev1MSL ,

I'm not sure I understand the issue. When loading a reference mesh from an .obj file, we respect the normals in the file if present: https://github.com/NVlabs/nvdiffrec/blob/main/render/obj.py#L115

For the mesh we optimize, we run auto_normals in each training step: https://github.com/NVlabs/nvdiffrec/blob/a3e73909a01887c8a135235ff860dd23a045cc1b/geometry/dlmesh.py#L42

Kev1MSL commented 1 month ago

Alright I think I understand better now, but I am not sure why you take the normal from the file, is it for supervision when you train your model?