Does is support derivatives with respect to the extrinsic parameters?

[huameichen0523]

Just wondering if it is possible to fit a mesh to a depth image by changing extrinsic parameters of the camera? In other words, the object is rigid rather than deformable. If yes, can you provide an example? Thank you

[martinResearch]

In the camera projection backward pass I don't compute derivatives w.r.t the cameras extrinsic parameters ( in project_points_backward I compute only derivatives w.r.t the points positions points_3d_b) so we can't fit the mesh by changing the camera extrinsic parameters using this "hand coded" derivative backpropagation function as we don't have gradients for the extrinsic parameters.

However instead of fitting the mesh by moving the camera by changing it extinsic parameters , you can move the mesh itself , which is what I do in the example https://github.com/martinResearch/DEODR/blob/master/deodr/examples/depth_image_hand_fitting.py. In order to optimize over the rotation you need a representation that does not have singularities , which is why I use quaternions (that I normalize after each step). You can make the fitting rigid by commenting out the line https://github.com/martinResearch/DEODR/blob/8aa25cd442db080c7c50fde1534956ac9b0d6c68/deodr/mesh_fitter.py#L169 (and then remove or comment part of code above it that becomes unused)

Alternatively, if you use either tensorflow or pytorch , then you can potentially get derivatives on the extrinsic parameters using the automatic differentiation they provide, in which case you could optimize over the extrinsic parameters. You will need to make sure the 3x3 submatrix of the extrinsic matrix corresponding to the rotation remains a rotation matrix through the iterative fitting updates (using a quaternion reparamterization for example or by reprojecting on the 3D rotation group after each step of the gradient descent, or moving along a geodesic on the 3D rotation group using exponential maps).