Mismatch of HD and SD while calibrating two sensors extrinsically

[jdeeee]

Note: I have made some edits on 02.02.18 10:45, because some statements in my original post were simply wrong.

Hey,

following task: I am trying to calibrate two Kinect v2 sensors extrinsically using a checkerboard. To do this, I am using a tool called open_pTrack, but I do not think that the problem is in their tool.

observations: I have conducted several tests, because I noticed an unexplainable offset in one of my tests. In all tests I set the depth_shift in calib_depth.yaml to 0, since the calibrated value I got worsened the results. The tests tell you on which RGB images I calibrated (SD or HD) and if the used camera was calibrated using kinect2_calibration or not. The offset/shift I am referring to means that the world origin is actually not placed on the correct checkerboard location.

checkerboard calib on SD-RGB-images & camera uncalibrated: → works for SD point cloud, HD point cloud is shifted (expected behavior. In the latter case there is no calibration) SD & camera calibrated: → SD-Cloud & HD-cloud: pretty correct. (expected behavior) HD & Camera uncalibrated →SD-PC: quite big shift, HD-PC: big offset on Z axis. Offset/shifts are not equal. (this is expected, because there is no HD-SD calibration available, the color on the HD-PC would also be wrong). HD & Camera calibrated → SD-PC: ~2cm offset <==> HD-PC: ~2cm offset, seems to be the same. (unexpected, where does this offset come from?? Maybe the HD-image is somewhat shifted, but you can’t really see it in the HD-pointCloud? Maybe depth to color works, but color to depth not?)

So, the issue is: When I use HD-RGB images for extrinsic calibration, the resulting point clouds will have an offset to the world origin. Why?

I thought this issue might be related to #160 and #173 even though I believe I do not have the problem of misaligned sd and hd point clouds, if I understood their problem correctly.

[JimmyDaSilva]

Hi @jdeeee for coming out on this issue as well. We have exactly the same problem here.

@wiedemeyer Let's put my extrinsics calibration to the robot aside. Thinking more about this I came out with a much more self explanatory demonstration of the problem.Anyone can try this, because it doesn't require a robot.

Just display the HD and SD pointclouds at the same time with different colors on RVIZ. There is definitely a shift between the two. And it's between 5 and 10cm

[jdeeee]

Hi!

I thought so too, but I am not so sure anymore. Because, as I stated, I can't observe the mentioned offset when I plot the hd and sd cloud in the same frame (maybe its a bit difficult to see on the picture): @JimmyDaSilva: QHD and HD should both be in the same frame. Do you experience the same issue, when choosing "kinectV2_rgb_optical_frame" as fixed frame under global options?

Edit: maybe its the same problem after all. You chose "world" as fixed frame, so you calibrated the Kinects with some calibration pattern using some images (either HD, QHD, or SD) as well. So most likely the respective point cloud (i.e., HD, QHD or SD) will be correct, the other ones will have an offset => same problem.

[JimmyDaSilva]

@jdeeee So we have different problems then. Same with HD or QHD. Changing fixed_frame on RVIZ won't change anything.

@wiedemeyer I still don't understand why I observe this shift. I remember recalibrating at least 3 times. Did you make any important changes in the calibration process in the last year ? I can't remember, do you use the factory internal offsets or the ones calculated during the kinect2_calibration for the distance between ir and rgb ?

Thanks ;)

[jdeeee]

Hey, I have made some big changes to my original post, since some of my observations were simply wrong. @JimmyDaSilva we seem to have different problems here, but maybe #173 is still related...

@wiedemeyer Are you still working on this package and do you have an idea what is going on?

[SiddhantNadkarni]

Having same issue. Errors in calibration when I use HD images from Kinect itself, or even an external camera, there is a shift in world origin. Please let me know if anyone has managed to resolve this issue. Thanks