search

IntelRealSense / realsense-ros

ROS Wrapper for Intel(R) RealSense(TM) Cameras
http://wiki.ros.org/RealSense
Apache License 2.0
2.58k stars 1.76k forks source link

Different calibration data on ros topic and device #750

Closed eirikaso closed 4 years ago

eirikaso commented 5 years ago

The calibration data that is published on the topic /camera/infra(1/2)/camera_info is different from what is found when it is read directly from the camera using "Intel.Realsense.CustomRW -r". Why is this, and which is the correct one?

Camera: D435 OS: Ubuntu 18.04 ROS: Melodic CustomRW version: 2.6.8.0

/camera/infra1/camera_info output:

header: seq: 30 stamp: secs: 1556613706 nsecs: 72526217 frame_id: "camera_infra1_optical_frame" height: 800 width: 1280 distortion_model: "plumb_bob" D: [0.0, 0.0, 0.0, 0.0, 0.0] K: [642.3675537109375, 0.0, 643.1593017578125, 0.0, 642.3675537109375, 404.2344970703125, 0.0, 0.0, 1.0] R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0] P: [642.3675537109375, 0.0, 643.1593017578125, 0.0, 0.0, 642.3675537109375, 404.2344970703125, 0.0, 0.0, 0.0, 1.0, 0.0] binning_x: 0 binning_y: 0 roi: x_offset: 0 y_offset: 0 height: 0 width: 0 do_rectify: False

Intel.Realsense.CustomRW output:

CustomRW for Intel RealSense D400, Version: 2.6.8.0

Device PID: 0B07 Device name: Intel RealSense D435 Serial number: 827312072319 Firmware version: 05.11.01.00

Calibration parameters from the device: resolutionLeftRight: 1280 800

FocalLengthLeft: 640.590088 640.183533 PrincipalPointLeft: 637.286316 402.818481 DistortionLeft: -0.055629 0.063853 -0.000203 -0.000965 -0.020792

FocalLengthRight: 642.577454 642.286743 PrincipalPointRight: 648.905396 406.992188 DistortionRight: -0.056637 0.067119 -0.000290 -0.001725 -0.022515

RotationLeftRight: 0.999958 -0.002542 -0.008774 0.002557 0.999995 0.001685 0.008770 -0.001707 0.999960 TranslationLeftRight: -49.850357 -0.108120 -0.091810

HasRGB: 1

resolutionRGB: 1920 1080

FocalLengthColor: 1383.647949 1384.156250 PrincipalPointColor: 949.643860 544.092773 DistortionColor: 0.000000 0.000000 0.000000 0.000000 0.000000 RotationLeftColor: 0.999849 -0.016204 -0.006292 0.016204 0.999869 0.000023 0.006291 -0.000125 0.999980 TranslationLeftColor: 14.810003 0.261981 0.088597

schmidtp1 commented 5 years ago

Hi @eirikaso, when you try rs-enumerate-devices -c, do you see the same?

eirikaso commented 5 years ago

Hi @schmidtp1 The intrinsic values from rs-enumerate-devices -c are the same as those published on the camera_info topic. Also the camera matrix looks the same on these two while it's different on "Intel.Realsense.CustomRW -r"

I do not have the exact same parameters on the camera any more so I will post the new ones with the values from rs-enumerate-devices -c included:

/camera/infra1/camera_info:

header: seq: 24 stamp: secs: 1557919254 nsecs: 636695623 frame_id: "camera_infra1_optical_frame" height: 800 width: 1280 distortion_model: "plumb_bob" D: [0.0, 0.0, 0.0, 0.0, 0.0] K: [639.0849609375, 0.0, 644.4653930664062, 0.0, 639.0849609375, 404.2340393066406, 0.0, 0.0, 1.0] R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0] P: [639.0849609375, 0.0, 644.4653930664062, 0.0, 0.0, 639.0849609375, 404.2340393066406, 0.0, 0.0, 0.0, 1.0, 0.0] binning_x: 0 binning_y: 0 roi: x_offset: 0 y_offset: 0 height: 0 width: 0 do_rectify: False

Intel.Realsense.CustomRW -r:

CustomRW for Intel RealSense D400, Version: 2.6.8.0

Device PID: 0B07 Device name: Intel RealSense D435 Serial number: 827312072319 Firmware version: 05.11.01.00

Calibration parameters from the device: resolutionLeftRight: 1280 800

FocalLengthLeft: 640.590088 640.183533 PrincipalPointLeft: 637.286316 402.818481 DistortionLeft: -0.055629 0.063853 -0.000203 -0.000965 -0.020792

FocalLengthRight: 642.577454 642.286743 PrincipalPointRight: 648.905396 406.992188 DistortionRight: -0.056637 0.067119 -0.000290 -0.001725 -0.022515

RotationLeftRight: 0.999975 -0.002564 -0.006569 0.002552 0.999995 -0.001754 0.006574 0.001738 0.999977 TranslationLeftRight: -49.850437 -0.109076 0.017740

HasRGB: 1

resolutionRGB: 1920 1080

FocalLengthColor: 1383.647949 1384.156250 PrincipalPointColor: 949.643860 544.092773 DistortionColor: 0.000000 0.000000 0.000000 0.000000 0.000000 RotationLeftColor: 0.999797 -0.018567 -0.007772 0.018558 0.999827 -0.001230 0.007793 0.001086 0.999969 TranslationLeftColor: 14.809196 0.296854 0.110884

rs-enumerate-devices -c: (Only intrinsics of "infrared1" and extrinsics from "infrared1" to "infrared2")

Intrinsic of "Infrared 1" / 1280x800 / {Y8}
Width: 1280
Height: 800
PPX: 644.465393066406
PPY: 404.234039306641
Fx: 639.0849609375
Fy: 639.0849609375
Distortion: Brown Conrady
Coeffs: 0 0 0 0 0

Extrinsic from "Infrared 1" To "Infrared 2" : Rotation Matrix: 1 0 0 0 1 0 0 0 1

Translation Vector: -0.0498505607247353 0 0

cichard commented 5 years ago

Hi @schmidtp1 The intrinsic values from rs-enumerate-devices -c are the same as those published on the camera_info topic. Also the camera matrix looks the same on these two while it's different on "Intel.Realsense.CustomRW -r"

I do not have the exact same parameters on the camera any more so I will post the new ones with the values from rs-enumerate-devices -c included:

/camera/infra1/camera_info:

header: seq: 24 stamp: secs: 1557919254 nsecs: 636695623 frame_id: "camera_infra1_optical_frame" height: 800 width: 1280 distortion_model: "plumb_bob" D: [0.0, 0.0, 0.0, 0.0, 0.0] K: [639.0849609375, 0.0, 644.4653930664062, 0.0, 639.0849609375, 404.2340393066406, 0.0, 0.0, 1.0] R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0] P: [639.0849609375, 0.0, 644.4653930664062, 0.0, 0.0, 639.0849609375, 404.2340393066406, 0.0, 0.0, 0.0, 1.0, 0.0] binning_x: 0 binning_y: 0 roi: x_offset: 0 y_offset: 0 height: 0 width: 0 do_rectify: False

Intel.Realsense.CustomRW -r:

CustomRW for Intel RealSense D400, Version: 2.6.8.0 Device PID: 0B07 Device name: Intel RealSense D435 Serial number: 827312072319 Firmware version: 05.11.01.00 Calibration parameters from the device: resolutionLeftRight: 1280 800 FocalLengthLeft: 640.590088 640.183533 PrincipalPointLeft: 637.286316 402.818481 DistortionLeft: -0.055629 0.063853 -0.000203 -0.000965 -0.020792 FocalLengthRight: 642.577454 642.286743 PrincipalPointRight: 648.905396 406.992188 DistortionRight: -0.056637 0.067119 -0.000290 -0.001725 -0.022515 RotationLeftRight: 0.999975 -0.002564 -0.006569 0.002552 0.999995 -0.001754 0.006574 0.001738 0.999977 TranslationLeftRight: -49.850437 -0.109076 0.017740 HasRGB: 1 resolutionRGB: 1920 1080 FocalLengthColor: 1383.647949 1384.156250 PrincipalPointColor: 949.643860 544.092773 DistortionColor: 0.000000 0.000000 0.000000 0.000000 0.000000 RotationLeftColor: 0.999797 -0.018567 -0.007772 0.018558 0.999827 -0.001230 0.007793 0.001086 0.999969 TranslationLeftColor: 14.809196 0.296854 0.110884

rs-enumerate-devices -c: (Only intrinsics of "infrared1" and extrinsics from "infrared1" to "infrared2")

Intrinsic of "Infrared 1" / 1280x800 / {Y8} Width: 1280 Height: 800 PPX: 644.465393066406 PPY: 404.234039306641 Fx: 639.0849609375 Fy: 639.0849609375 Distortion: Brown Conrady Coeffs: 0 0 0 0 0

Extrinsic from "Infrared 1" To "Infrared 2" : Rotation Matrix: 1 0 0 0 1 0 0 0 1 Translation Vector: -0.0498505607247353 0 0

Hi @eirikaso ! I have got the same question, do you have feed back on this ?

eirikaso commented 5 years ago

Hi @cichard Not really. I am very interested in finding it out though, as it seems really strange to me. I for instance do not understand how the rotation matrix on ROS can be an identity matrix when it is not on the realsense camera. However, when I use the method "rs-enumerate-devices -c", the rotation matrix left to right is an identity matrix. @schmidtp1 , do you have any idea?

I have looked a bit into it and it seems that opencv projects the two rectified images onto a new virtual image plane. This might have something to do with it but I have not been able to get to the bottom of it.

Please update if you make any discoveries...

cichard commented 5 years ago

On my side, I have the same result as you when i use the rs-enumerate-devices -c i have the identity matrix and in ROS topic identity matrix too. But when i use Intel.Realsense.CustomRW -r I have something like : RotationLeftRight: 0.999985 0.005334 -0.001527 -0.005333 0.999985 0.000757 0.001531 -0.000748 0.999999 TranslationLeftRight: -49.925358 0.034362 0.183151

i have found something quite interesting on intrinsics on this wiki They say that the stream is rectified by default

D400-Series

Left and right infrared images are rectified by default (Y16 format is not)

The two infrared streams have identical intrinsics
The two infrared streams have no distortion
There is no rotation between left and right infrared images (identity matrix)
There is translation on only one axis between left and right infrared images (translation[1] and translation[2] are zero)
Therefore, the y component of pixel coordinates can be used interchangeably between these two streams
eirikaso commented 5 years ago

Yes, that is correct that the images are rectified by default. I am however using the camera under water and wanted to use a different calibration technique with a large chessboard as I believe that will give better results.

I have now used the uncalibrated Y16 stream to calibrate and passed in the parameters to the realsense camera. The rotation matrix I sent was NOT an identity matrix, but I still get an identity matrix when I read them back using rs-enumerate-devices -c.

Is it so that rs-enumerate-devices -c show the parameters that are used in rectification on the camera? This rectification is done with the assumption that there is not rotation between the IR cameras and only translation on one axis, and the values returned are therefore adjusted?

cichard commented 5 years ago

@eirikaso how did you pass them :

eirikaso commented 5 years ago

I passed it with an xml file. I modified the ros-image-pipeline camera-calibration package to write results to an xml file that is similiar to the one used by realsense. I then passed it using Intel.Realsense.CustomRW

cichard commented 5 years ago

What I guess but not sure is exactly what you said : parameters are used internally so when you use them inside ROS, they are already rectified

eirikaso commented 5 years ago

Yes, but the Y16 format is not rectified. This is what I used for calibration

schmidtp1 commented 5 years ago

@doronhi, do you have any ideas?

RealSenseSupport commented 4 years ago

Hi @eirikaso,

The issue is currently under investigation.Updates will be posted as soon as investigation findings become available.

Thank you!

doronhi commented 4 years ago

Sorry for popping in so late. I'm not I understand the issue: The transformation in topic "/camera/infra1/camera_info" refers to the position of sensor "infra1" relative to the depth sensor. Since in the D400 series the depth is built in infra1 coordinate system they are the same and hence the unit transformation. The transformation shown in the output of CustomRW:

RotationLeftRight: 0.999958 -0.002542 -0.008774
0.002557 0.999995 0.001685
0.008770 -0.001707 0.999960
TranslationLeftRight: -49.850357 -0.108120 -0.091810

relates to the position of leftIR (infra1) relative to rightIR (infra2). They are not the same sensor so we have translation and some rotation.

eirikaso commented 4 years ago

It's been a while so I don't remember exactly but I think it is OK.

I was confused that the intrinsics like distortion etc was not the same on the realsense device and published on the ros topic. I see now that the camera stream published on ros is already calibrated using the realsense calibration and distortion parameters, and these are therefore different when distributed on the calibrated topic.