Possible bug in image projection

[marip8]

I came across a possible bug while developing a simulated observation generator in which the 2D calibration algorithms do not check whether the z value of the target point, projected into the camera frame, is greater than zero (i.e. actually in front of the camera). See this file for reference.

The issue here is:

• User provides a bad initial guess for the target or camera position (probably on accident)
• Some target observations show up behind the camera due to the bad guess
• The image projection math is no longer meaningful
• The optimization does not converge

I suggest we check that the z-value of the 3D target point in camera coordinates is greater than zero and throw an exception indicating that the initial transform guess was probably incorrect. I think this would be more helpful than the calibration silently failing to converge

@schornakj @drchrislewis is this a reasonable interpretation of this issue?

[marip8]

Here are a couple images of the simulated observations I made that brought this problem to light. The points with small normals represent camera locations and camera Z-axes. The single point with the larger normal is the target guess and target guess Z-axis.

Good

The target is always in front of the camera positions

Bad

The target is behind several camera positions and thus not technically visible

[schornakj]

I think that the pinhole camera model actually does still work when reprojecting object points that are behind the image plane, although this situation of course isn't feasible with real optics.

Seems like the core issue here is that if an object point is initially on the wrong side of the camera plane due to inaccurate initial values for transforms, the optimization tries to converge onto an incorrect local minimum. I suspect that strange discontinuous things happen to the residual as the reprojected point "flips" from -Z to +Z, which probably is a factor in preventing the optimization from converging.

We could check the sanity of initial guesses during a problem setup validation step. I'd avoid doing this check inside the cost function.

[drchrislewis]

@schornakj @marip8 This is a well understood problem. The ROS calibration does not use initial guesses from the user. Instead, there are well known methods for computing a reasonable PnP solution analytically from just a few observations. Then they let the optimization refine it. I never found this necessary because I always per-arrange the poses to the neighborhood of the solution. Even when I do robot driven intrinsic calibration, I use the forward kinematics and approximate target pose (grabbed from TF) to initialize the optimization. Having to add a layer of PNP code complicates things for a generic observation class where the cost being employed may change. If you want to do the PNP, look for papers under "calibration" Ax=B. An old office mate of mine at Purdue did his PhD on that. I didn't know what it was at the time, but now I do.

[marip8]

Distilling these comments:

• Sanity check the initial positions of the target and camera during problem set up
  • Raise an error if the observations can't all be seen correctly
• Don't throw an error in the optimization itself
• Using a PnP optimization to determine an initial guess
  • Possibly create a function/library to do this
  • Include in higher level code (i.e. creation of the calibration data/problem)

[marip8]

Addressed by #38 here; closing