search

NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.37k stars 146 forks source link

handling gradients between calling nvdiffrast functions #82

Closed maorp closed 2 years ago

maorp commented 2 years ago

thanks for the awesome work. a question about nvdiffrast gradients, how should i handle the gradients when i do some operations between calling nvdiffrast functions, like in this 2 view texture projection code, here proj_coords created after interpolation and then W divided and normalized to [0,1] using torch, but then i call dr.texture and pass only the interpolation gradients to it. should I explicitly pass also torch grads to dr.texture?

triangles = base_mesh.t_pos_idx.int()
pos_src = ru.xfm_points(base_mesh.v_pos[None, ...], cam_mats[src_id], use_python=True) 
pos_trg = ru.xfm_points(base_mesh.v_pos[None, ...], cam_mats[trg_id], use_python=True)
rast_trg, grads_trg = dr.rasterize(glctx, pos_trg, triangles, resolution=image_sizes[trg_id])
proj_coords_homogenic, diffs_attr = dr.interpolate(pos_src[...,[1,0,3]], rast_trg, triangles, rast_db=grads_trg, diff_attrs='all') 

proj_coords = proj_coords_homogenic[...,[1,0]] / proj_coords_homogenic[...,[2,2]] # divide in w coord, now in [-1,1]
proj_coords = (proj_coords + 1)/2 # normalize to [0,1] (for dr.texture() space)
proj_coords[torch.isnan(proj_coords)] = 0

rendered = dr.texture(images[src_id][None,...], proj_coords, filter_mode='linear', uv_da=diffs_attr)
s-laine commented 2 years ago

The screen-space derivatives (here diffs_attr) should be consistent with the uv coordinates used for lookups -- otherwise mipmapping won't work as intended, and wrong mip levels may be used. Note that mipmapping is enabled only if the filter mode is either linear-mipmap-nearest or linear-mipmap-linear. In your code, you appear to have mipmapping turned off, so the contents of diffs_attr don't actually affect the results at all.

If you want to use mipmapped texture lookups, you'll probably get a good approximation for the screen-space derivatives by dividing diffs_attr by 2*W before the lookup, to account for the division and scale you do for the uv coordinates. It's not exactly correct, as W isn't constant in pixel space, but I doubt it's necessary to take that into account.