search

NVlabs / nvdiffmodeling

Differentiable rasterization applied to 3D model simplification tasks
Other
455 stars 30 forks source link

matrix gradients does not flow back when using renderutils xfm_points #11

Closed maorp closed 2 years ago

maorp commented 2 years ago

hi, when using renderutils.xfm_points (e.g. in render.py v_pos_clip = ru.xfm_points(mesh.v_pos[None, ...], mtx_in) - and trying to optimize camera pose in mtx_in - gradients does not flow back. Only when using use_python=True it worked.

jmunkberg commented 2 years ago

Yes, unfortunately, the Cuda kernels for those transform operations only provide gradients to the positions currently, not to the matrix (that was all we needed for our use case). We should probably add that limitation to the docs, thanks for pointing that out!
The implementation of those kernels is here: https://github.com/NVlabs/nvdiffmodeling/blob/main/src/renderutils/c_src/mesh.cu

So for now, if you need gradients to the matrix, please write the transform op in PyTorch (or use the use_python=True flag as mentioned above). If should be straightforward to add gradients to the matrix as well, if this op is a performance bottleneck, but we haven't got around to it.