Closed marip8 closed 4 years ago
Here are a couple images of the simulated observations I made that brought this problem to light. The points with small normals represent camera locations and camera Z-axes. The single point with the larger normal is the target guess and target guess Z-axis.
The target is always in front of the camera positions
The target is behind several camera positions and thus not technically visible
I think that the pinhole camera model actually does still work when reprojecting object points that are behind the image plane, although this situation of course isn't feasible with real optics.
Seems like the core issue here is that if an object point is initially on the wrong side of the camera plane due to inaccurate initial values for transforms, the optimization tries to converge onto an incorrect local minimum. I suspect that strange discontinuous things happen to the residual as the reprojected point "flips" from -Z to +Z, which probably is a factor in preventing the optimization from converging.
We could check the sanity of initial guesses during a problem setup validation step. I'd avoid doing this check inside the cost function.
@schornakj @marip8 This is a well understood problem. The ROS calibration does not use initial guesses from the user. Instead, there are well known methods for computing a reasonable PnP solution analytically from just a few observations. Then they let the optimization refine it. I never found this necessary because I always per-arrange the poses to the neighborhood of the solution. Even when I do robot driven intrinsic calibration, I use the forward kinematics and approximate target pose (grabbed from TF) to initialize the optimization. Having to add a layer of PNP code complicates things for a generic observation class where the cost being employed may change. If you want to do the PNP, look for papers under "calibration" Ax=B. An old office mate of mine at Purdue did his PhD on that. I didn't know what it was at the time, but now I do.
Distilling these comments:
I came across a possible bug while developing a simulated observation generator in which the 2D calibration algorithms do not check whether the z value of the target point, projected into the camera frame, is greater than zero (i.e. actually in front of the camera). See this file for reference.
The issue here is:
I suggest we check that the z-value of the 3D target point in camera coordinates is greater than zero and throw an exception indicating that the initial transform guess was probably incorrect. I think this would be more helpful than the calibration silently failing to converge
@schornakj @drchrislewis is this a reasonable interpretation of this issue?