search

code-iai / iai_kinect2

Tools for using the Kinect One (Kinect v2) in ROS
Apache License 2.0
873 stars 519 forks source link

SD image and cloud seem shifted #173

Open JimmyDaSilva opened 8 years ago

JimmyDaSilva commented 8 years ago

On #160 I was talking about a shift on the SD cloud, but it seems that the images are shifted too. This is a problem for me because I use the IR to calibrate the kinectv2.

See below that the robot is well placed on the SD image but not the HD/QHD

HD hd QHD qhd SD sd

kohrt commented 8 years ago

Hi, since only qhd and hd are wrong, it's probably a not optimal calibration. How big where the error values that where outputted by the calibration tool?

JimmyDaSilva commented 8 years ago

I can't remember. Is there a way to get these values back ? What's the unit of the returned cv::stereoCalibrate error? What approximate error value would mean calibration is correct ?

Thanks!

kohrt commented 8 years ago

if you still have the recorded images you could just run the calibration commands again.

JimmyDaSilva commented 8 years ago

I went through a whole calibration again and made sure the rms error was not to big. These are the error values I got:

The translation between ir_optical_frame and rgb_optical_frame is about 0.052 as expected. I have 0.051789.

But unfortunately there is still a shift. You can see on the following images that :

Extrinsic calibration of the Kinect2 pose was done with vision markers using the IR image, that's why it is aligned to the SD image.

HD hd QHD qhd SD sd

kohrt commented 8 years ago

The reprojection errors are looking fine. For qhd/hd the calibration is used. sd uses the sensor internal calibration. Therefore the calibration only effects qhd/hd, but not sd. Actually the images should be matching as good as the point clouds from the same topic group. You should probably see the same misalignment between the arm and the cloud, when you move the the rviz camera to the position and orientation of the kinect2_rgb_optical_frame for qhd/hd or kinect2_ir_optical_frame for sd. Another source of error could be the arm to camera calibration. And 14 days ago libfreenect got updated, they removed the hard coded tables and are now calculating these values based on the internal sensor data. If you did the calibration before this update but updated libfreenect2 in the meantime, than this could also result in misalignment. Maybe you have to redo the whole calibration in that case.

JimmyDaSilva commented 8 years ago

Any way to use the calibration for the sd too ?

I totally agree with you that the misalignment should be the same on both the pointcloud and the image. That's what I find really strange.

As you can see the arm to camera calibration is good on the sd image or hd cloud ...

Last time I calibrated was the day I last posted, and that was 7 days ago.

Did you get a chance to try to reproduce this problem on your side ?

kohrt commented 8 years ago

No, it is not possible to use the calibration results for the sd. But could you take a screenshot with the rviz view aligned to the kinect2_rgb_optical_frame, so that I can see the point cloud and the color image from the same perspective? Did you also used the most recent version of libfreenect2 back then? Right now I don't see a way to reproduce it here easily.

JimmyDaSilva commented 8 years ago

I won't be able to interact with the robot until Monday but I will give you whatever you need then.

JimmyDaSilva commented 8 years ago

Sorry! Kind of forgot about this.

The point of view is exactly the same on both images. It's easier to see the shift if you download the images and switch from one to the other with your favorite image viewer.

QHD qhd SD sd

JimmyDaSilva commented 8 years ago

@wiedemeyer Any updates on this issue ?

kohrt commented 8 years ago

@JimmyDaSilva I though about it a couple of times but I have no real updates. The problem is, that the projected robot arm overlay does not fit the arm on the camera image / point cloud. The cause could be the camera calibration, or the arm to camera calibration, the arm calibration, or even the arm itself. Since the camera calibration seams to be fine, you should look into the arm to camera calibration. How did you calibrated the arm and the camera? Sometimes it might help to print the projected chessboard corners onto the images used for the calibration to see if there is some general error.

JimmyDaSilva commented 8 years ago

@wiedemeyer I have ran all calibrations, intrinsics, extrinsics a couple times since then and I still see the same error. It still doesn't make sense to me that the SD pointcloud would align with the robot on the image and not on the HD one, or the other way around.

What all this means is that the SD image/cloud published on _ir_optical_frame and the HD image/cloud published on _rgb_optical_frame don't align. So there is only two possibilities:

Are you using the extrinsics calibration computing the transformation between the two frames or the default value of about 5cm ?

kohrt commented 8 years ago

Yes the calibrated extrinsic parameters are used for the transform The HD topics are using the calibrated intrinsic and extrinsic parameters, and the sd topic uses the sensor internal parameters. This could be a reason why SD and HD do not match perfectly. But on the images also the SD image seams to be off (but less then the HD images), therefore I would look at the arm to camera calibration.

JimmyDaSilva commented 8 years ago

Calibration to SD image is like 1cm accurate, I don't need anything better than that. No problem with the arm to camera calibration here

How should I proceed to try using calibrated parameters for the sd topics?

There has to be a problem somewhere when the SD topic is created because the shift is obvious: http://gifmaker.cc/PlayGIFAnimation.php?folder=20160104203OeCWwj8Pc1kHFHOomP7pQ&file=output_JLI0pb.gif

kohrt commented 8 years ago

If you want you can use the calibrated parameters for the SD topic as well, you just need to paste the following lines after line 579:

cameraMatrixIr = cameraMatrixDepth;
distortionIr = distortionDepth;

But the registration of the color image relies on the build-in parameters, it will not work correctly with different ones.

If you want the robot to fit the HD images, you could try to calibrate it against those images, instead the SD images.

JimmyDaSilva commented 8 years ago

And so what about the color registration? How could we change it to use the calibrated parameters?

kohrt commented 8 years ago

I don't know.

TrinhNC commented 7 years ago

@JimmyDaSilva Have you solved this problem? I'm facing the same issue.

JimmyDaSilva commented 7 years ago

Hi @TrinhTUHH. I have not looked in the code to find the issue but there is definitely an error somewhere. For now I have hacked the problem by switching the frame manually to rgb instead of depth and vice-versa.

@wiedemeyer Did you look into this ? I pretty sure something is wrong somewhere which is not calibration.