search

NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.29k stars 139 forks source link

back-project #137

Closed XuYunqiu closed 9 months ago

XuYunqiu commented 9 months ago

Hi, thanks for the great work. I'm wondering is there any way to gather pixels on 2D image and back-project them onto 3D vertex/faces (like inverse render pipeline)?

s-laine commented 9 months ago

Projecting a reference image onto the texture of a mesh has been discussed in issue #1. Projecting onto vertices or faces may be a bit more complicated, so if texture works for you, that would probably be easiest.

XuYunqiu commented 9 months ago

Thanks for the prompt reply and sorry for the ambiguity. I'm not familiar enough with some details of the render process. In my case, I have the index of some pixels on the 2D image plane, and I want to get the corresponding 3D faces.

In my understanding, the rast[:, x, y, 3] is the corresponding face (i.e., triangle_id) of the pixel (x, y) in the rendered image, where rast, rast_db = peeler.rasterize_next_layer(). Is that correct?

s-laine commented 9 months ago

That's correct, except you have x and y coordinates flipped. Also note that the value is offset by one, so you need to subtract 1 from the value in rast[:, y, x, 3] to get the triangle ID. A value of 0 means there was no triangle in that pixel.

XuYunqiu commented 9 months ago

got it, thanks again for your kind reminder