Render the final result?

[soolone]

Hello. First of all, thank you very much for your excellent work. I'm sorry I'm not very familiar with graphics engines, but now I want to render the final output, such as hotdog's Mesh. There are some questions below hope to get your answer: 1.According to the description in the article, kd.png\ks.png\kn.png describes the material of the model，But the final output has.mtL file, is.mtL equivalent to three PNG files？Now when I use mesh.load_mesh(mesh.obj), the api will directly use mesh.mtl instead of loading three Png files

2. If.mtL means the same as three Png files, I can render by loading the model and loading Png files?

[jmunkberg]

Thanks @soolone ,

Please see the included examples for example .mtl files. We abuse the .mtl format a bit and specify the bsdf

Example from https://github.com/NVlabs/nvdiffrec/blob/main/data/bob/bob_tri.mtl with added comments

newmtl feathers  # Material name
bsdf pbr  # Shader type, this is our customization, not part of standard .mtl format
Ks 0.00 0.25 0.00 # specular parameters :  x(ignored) , linear roughness, metalness (ORM format)
map_Kd bob_diffuse.png # albedo map
map_Ks specular_map.png # specular map on ORM format, O channel is ignored
bump filename.png # tangent space normal map

You can also load the models after optimization in Blender using something like this https://github.com/NVlabs/nvdiffrec/issues/27

[soolone]

Thanks, Problem solved

[soolone]

@jmunkberg I used Euler Angle as rotation vector control during rendering, but this inevitably resulted in the problem of universal locking, I notice there is a function called lookAt in util.py. Is it modeled in polar coordinates? What is the meaning of each parameter?

[jmunkberg]

For a description of lookAt, see slide 4-11 here: https://fileadmin.cs.lth.se/cs/Education/EDA221/lectures/latest/Lecture5_web.pdf

[soolone]

Thank you. The problem has been solved successfully. @jmunkberg

[soolone]

Sorry to bother you again. I am now able to control the final render using the rotation(3 3) and translation(3 1) matrix, but your project uses Campos(3 1) and MVP(4 4) matrix for rendering(I looked at the source code and it seems that MVP represents the internal parameter matrix,Campos represents the external parameter matrix). Now I hope to get the corresponding (u, v) through the 3D key points (x,y,z) of the feature. Could you please provide me with the calculation method or idea through Campos and MVP? Or how can I get the correct inner parameter matrix K (3 * 3)? which allows me to use KRT to compute the correct (u,v) coordinates. @jmunkberg

[jmunkberg]

We rely on nvdiffrast, which uses OpenGL conventions for the camera transform. https://nvlabs.github.io/nvdiffrast/#coordinate-systems

To transform to OpenCV standards, we did write a loader for DTU (unfortunately not available in the public release). For that I used https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html (see Detailed Description) and the decomposeProjectionMatrix function https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#gaaae5a7899faa1ffdf268cd9088940248 and built an OpenGL projection matrix from that, see slide 25 here: https://fileadmin.cs.lth.se/cs/Education/EDA221/lectures/latest/Lecture5_web.pdf

[soolone]

@jmunkberg Thanks for your help, the problem has been solved. After calculating 3d coordinates using MVP，I also need to do normalization, clipping, and viewport calculations. Now it is possible to control the position of an object and the coordinates of a keypoints at arbitrary resolution using the camera position and nvdiffrast.