Pose optimizing algorithm using image based loss doesn't change pose on each iteration

[neanea04]

❓ Questions and Help

Description: I am trying to use the Theseus package to optimize the pose of an object from different 3D locations, so that the estimated pose from one camera observed from another camera is similar to the estimated pose from that camera. I am using a mean squared error (MSE) loss on the rendering of poses to optimize the pose.

However, I am experiencing an issue where the optimizing variables (v1, v2, v3, and v5) do not change on each iteration of the optimization. The initial pose given to the optimizer stays the same value for all the steps, and subsequently the LM optimization cannot find the optimized pose by minimizing the MSE error between two pictures.

I have tried running the optimization with different parameter values, but the optimizing variables still do not change. There are no error messages displayed during the optimization. t:

Operating system: Ubuntu 18.04 I would appreciate any help in resolving this issue. Thank you.

[luisenp]

Hi @neanea04, thanks for reporting. It's difficult to diagnose just by looking at code snippets. Have you checked if the jacobians computed by AutodiffCostFunction are non-zero? You can explore this by putting a breakpoint in this line.

You could also use

optimizer.objective.update(inputs)
jacobians, error = cost_function.jacobians()

Probably unrelated, but the recommended way to use our optimizers is via a TheseusLayer, as explained in our intro tutorial.

Also, you can reduce your profiling boilerplate with our Timer class (see example).

[neanea04]

@luisenp Thanks for the response :).The Jacobian seems to be zero vectors. Sorry I am new to optimization frameworks. As a simple explanation of my approach, I am just passing camera to object pose for few different cameras to the cost function. Using these poses inside the cost function I use my render function to render these poses. MSE loss between rendered images is used to optimize the camera pose. Is the jacobian zero because I am using a rendering function inside the cost function? The error output from the cost function seems to be alright.

[luisenp]

It looks like you are using a non-differentiable rendering function. I see you are doing some numpy conversions from the optimization variables, which would definite break the autodiff graph. You'll need a differentiable rendering function so that torch can compute the gradients automatically for you.

@mhmukadam @fantaosha are you familiar with any libraries that can be used here for computing the error with differentiable rendering? Maybe pytorch3d?

[mhmukadam]

Good catch @luisenp! Yes PyTorch3D has differentiable rendering, maybe also nerfstudio.

[neanea04]

Thank you for your suggestions @luisenp @mhmukadam . I will try that.