s-laine
commented
3 years ago You can project the input image into texture space using the existing primitives as follows.
First, project your 3D vertices into the reference image, yielding per-vertex 4D homogeneous NDC coordinates. Throw the z component away and call the remaining (x, y, w) vector as r_i. Then, rasterize your triangles into the texture atlas using their texture coordinates as positions and the r_i as interpolated attributes. This "unfolds" the mesh into texture space, yielding a texture atlas where each texel has the homogeneous image coordinate of the corresponding point in the reference image.
In each texel, divide the interpolated (x, y) components by interpolated w. Now your texture atlas has, for each texel, a 2D location from which to read the reference image. Remap these image-space coordinates from [-1, +1] to [0, 1] to get texture coordinates for addressing the reference image, and perform a texture sampling operation using the reference image as the input texture. This pulls the pixel colors from the reference image into texture space.
Note that this process also gives colors for texels that are occluded from the camera viewpoint. If you need to prevent this, you should rasterize the mesh in a standard way from camera to resolve visibility and do a shadow mapping -like test per texel to check if it is visible from the camera.
yxie20
Thank you for releasing this powerful tool!
In our research work, we are hoping to leverage nvdiffrast for joint texture and geometry optimization. I would love to hear your comments on whether incorporating projective texture mapping into nvdiffrast is a feasible task?
Given posed RGB images, depth maps, and a texture-less mesh, we hope to use projective texturing to initialize the mesh texture. The key idea depends on projection of posed camera images onto the mesh and then measuring the amount of visual discrepancy between the projected textures.
Thank you!