IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.56k stars 4.82k forks source link

L515 Extrinsic Calibration Between IMU, Camera, and Depth #10456

Closed Cpeck71 closed 2 years ago

Cpeck71 commented 2 years ago

Camera Model | L515 -- | -- Firmware Version | 01.05.08.01 Operating System & Version | Linux - Ubuntu 18.04 Kernel Version (Linux Only) | 4.9.253-tegra Platform | Jetson Nano SDK Version | 2.49.0 Language | C++ Segment | Computer vision/Navigation

Issue Description

Hello RealSense Team,

I would like some clarification on the extrinsic parameters between the hardware components of the L515. For refrence, I've looked at issue #8249 and the linked extrinsics documentation and OpenCV pinhole camera model pages from that comment. From my own work, it is my understanding that the published data from the IMU agrees with the documentation and the IMU frame is oriented as the blue frame drawn in the figure below (I hope it's okay I'm using a screen grab from the datasheet).

To my issue: I run the rs-enumerate-devices tool with the '-c' option to get the calibrated extrinsic parameters between all of the hardware components (IMU, depth and camera). The trouble i have is that the extrinsic translation vector doesn't agree with the frames drawn here. The IMU and Depth camera are perfectly aligned, and the depth to camera (along with IMU to camera) rotation matrices are so close to identity that for the sake of this argument lets consider them to be nearly aligned such that all hardware frames have the same orientation.

Now, according to the rs-enumerate-devices the translation vector from Gyro to Depth is: T_G2D = [0.01245,-0.01642,-0.00057] from Gyro to Camera: T_G2C = [0.01214,-0.00253,-0.0053];

If you note, the X distance is positive, however according to the drawing the IMU should be to the left of the Camera and Depth frames, i.e. going from Gyro to depth/camera should be a negative translation. Furthermore, the Camera to Depth translation is: T_C2D = [0.000307,-0.013997,0.004785]

which doesn't make sense either since it should be positive in the Y and negative in the Z (since the origin of the depth frame is 4.5mm recessed from the glass cover) according to my current understanding of the orientation of the hardware.

Is there something I'm fundamentally missing from this or possibly an error in the enumerate devices tool?

Thanks so much for the help, Caleb

RealsenseFrames

MartyG-RealSense commented 2 years ago

Hi @Cpeck71 With RealSense camera models that are in a casing and have front glass, the depth measurement is taken from the front glass and not the camera lenses behind the glass.

Adding -4.5 (mm) to the depth measurement to find the depth ground truth value at the lenses does not mean the depth will be a negative value, as it is subtracting 4.5 mm from the Z-depth measurement of the object or surface that is being observed by the camera.

In regard to from-front mechanical drawings of RealSense cameras it should be taken into consideration that from the camera user's perspective they will be behind the camera, and so the orientation of the imaging components will be reversed in the behind the camera perspective if the camera is rotated around so that the back of it is facing you.

The 400 Series stereo (left and right) depth cameras are an example of this, where the left imager is on the right -side of the camera on the frontal drawings.

Cpeck71 commented 2 years ago

Thank you Marty for the reply!

Talking about the component orientation, shouldn't it be dependent only on the orientation of the objects themselves (rather than the perspective from which the device is looked at)? Even if the X axis from the figure is flipped (to corrspond to me looking at it the other direction), what confuses me the most is what the T_C2D is negative in the Y direction. My understanding is that the extrinsic parameters should be in reference to their own frame (i.e camera to depth in the perspective of the camera frame), so that's why I'm a little confused at the output of the enumerate-devices tool.

Thank's so much for your time

Cpeck71 commented 2 years ago

Hello again, I was browsing other L500 issues and came upon this one. I realize this was from a couple of months ago, but was this an rs-enumerate-devices run from a L515? Because the values for "accel to color" are somewhat different than mine. Like what i have in the Z direction is given in the Y direction in the previous post. So, could it be something related to the original calibration? Thanks again

MartyG-RealSense commented 2 years ago

I do not think that your minus value on y translation for the color to depth extrinsic is wrong, as I have had the same in an L515 read-out from rs-enumerate-devices -c in a past case at https://github.com/IntelRealSense/librealsense/issues/9705#issuecomment-914202767

I have observed the same effect on the D435i camera model, where color to depth y translation is minus and depth to color y translation is positive. The read-out from that D435i can be found in an image at the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/1500000652922/comments/1500000554401

In regard to your latest comment about IMU to color: yes, that read-out would be from an L515 because I would test with the same camera that the person asking the question is using. It is normal for different cameras of the same model to have different values though, as the intrinsics and extrinsics of a particular camera are unique due to the manufacturing process in the factory.

Cpeck71 commented 2 years ago

Okay, the negative camera to depth Y distance seems a little unintuitive to me (like I said, my understanding is that the component frames are shown in the figure, so in my view it would be positive in the Y) but I can accept that for now.

I was looking at this issue and noticed that I was getting similar errors in the IMU where my norms are around 9.64 m/s^2. In my realsense-viewer I toggled the button form "correct IMU data" and I noticed a warning that effectively says "no IMU calibration found, using default intrinsic and extrinsic values". Do you recommend that I run through the calibration process? That might reveal something in the IMU extrinsics as well...

MartyG-RealSense commented 2 years ago

Yes, the IMU data correction button only works once an IMU calibration has been saved to the camera hardware using the Python calibration tool.

The IMU component in RealSense cameras does not have a built in calibration method and so the software tool is required to perform the calibration. Before a calibration is written, the Y Accel value may be around 9.4 to 9.6 instead of the ideal target of 9.80 (representing Earth's gravity). The Y Accel value also may keep changing even when the camera is stationary.

Once a calibration is applied, the correction button in the Viewer can 'fix' the IMU data so that Y is closer to the ideal 9.80 value.

There is certainly no harm in performing the IMU calibration if you want to improve the IMU values and I would recommend doing so.

Cpeck71 commented 2 years ago

I see, okay I'll run through the calibration and see what kind of results I get.

Does that IMU calibration affect the extrinsic parameters catalogued in the rs-enumerate-devices? I'm still a little misunderstood about the extrinsic parameters. Based on the frames being aligned, it shows that the IMU is in the -Y direction for both the camera and the depth frames, but I would think it would be +Y for the depth (below the IMU) and -Y for the camera.

MartyG-RealSense commented 2 years ago

A RealSense team member advises in https://github.com/IntelRealSense/librealsense/issues/5363#issuecomment-563421712 that the IMU translation vector is fixed and cannot be modified by the IMU calibration script.

I do not have information about the reason why the IMU is in the -Y direction, unfortunately.

Cpeck71 commented 2 years ago

Thanks for the link, I'll do the accel calibration today and see how that affects the output. Thanks also for the note on the extrinsic parameters being fixed, that makes sense why they would be. Is there any documentation about the calibration procedure, or some exact definitions for the extrinsic parameters as they are described? For example, the output says "From camera to depth" and I want to be sure that's either in the camera's perspective or the depth frame's perspective.

Thanks so much for your time and patience

MartyG-RealSense commented 2 years ago

The PDF version of the guide for the tool is the most detailed reference about the IMU calibration procedure, especially the IMU Calibration Parameters section on pages 8-9.

https://dev.intelrealsense.com/docs/imu-calibration-tool-for-intel-realsense-depth-camera

This section mostly focuses on the intrinsic parameters in detail though and suggests using librealsense tools to look at the extrinsics, which you have already done. The guide does offer the following paragraph about IMU extrinsics though.


The extrinsic parameters include:


Whilst 'Left IR' is a term usually associated with the 400 Series stereo depth cameras, the guide has extensive L515-specific information but does not name a specific model in the above information. This makes me think that the term 'Left IR' also applies to the single IR Transmitter component of the L515.

On the 400 Series cameras the depth origin is the center-line of the Left IR sensor, whilst on L515 the depth origin is the center of the IR Transmitter.

Cpeck71 commented 2 years ago

Hi Marty, I'm having some trouble with importing pyrealsense2 in order to use the IMU calibration tool. When im in the /librealsense/tools/imu-calibration directory i run python rs-imu-calibration.py and i get the error no module named pyrealsense2.

This morning I completely reinstalled the librealsense from source with RSUSB flag on and followed instructions from here except without the DBUILD_CUDA=true because I dont need to use cuda right now. Reading the instructions from the specific python wrapper, i see: update your PYTHONPATH environment variable to add the path to the pyrealsense library export PYTHONPATH=$PYTHONPATH:/usr/local/lib

Which i did inside the imu-calibration directory, but that doesnt help. Also note that I'm not inside the build folder when trying to run the python script, because inside the build folder of the python wrapper all i get are the .so files. Also also, there is no im-calibration directory in /build/tools directory.

Could you let me know if I missed a step? Thanks so much for all the help

Cpeck71 commented 2 years ago

EDIT: Nevermind, i needed to be using Python3 because the .so files had the extra bit in the name (eg. pyrealsense2.STUFF.so)

MartyG-RealSense commented 2 years ago

Thanks very much @Cpeck71 for the update about the .so file naming. Do you require further assistance with this case, please?

Cpeck71 commented 2 years ago

I plan to run a couple of tests tomorrow that should hopefully provide some more info to help understand my problem, so I'd like to leave the case open for just a couple of more days if thats okay

MartyG-RealSense commented 2 years ago

Yes, it is fine to leave the case open for a further time period. Good luck!

Cpeck71 commented 2 years ago

Hi Marty,

I've been doing some testing and still dont have much luck reconciling either my own calibrations or the extrinsic parameters from the L515. I'm starting to have concerns about the IMU's measurement quality and am curious if there may be something internally wrong with it. For example, below is output from a small test i did that essentially had the camera looking down and was traveling in a small circle. What's concerning is that the X and Y axes dont switch from positive to negative as it travels around, which makes me curious if there's a large bias or some other issue going on. I ran another test of simply lowering the camera at ~1m/s and was getting acceleration magnitudes of ~13 m/s^2 which is weird because, even with the gravity term, the motion was rather constant so I'd expect it to be pretty close to gravity.

AccelData

I retreive the IMU data through a callback directly from the IMU sensor (i.e i have an IMU sensor object and do IMU.start(IMUCallback) which retreieves the frame and therefore the data), and sometimes it can be very noisy, so I'm not sure if I have an error in my startup procedure.

I'm planning to run the rs-motion module tomorrow, so I'll report back if that is providing reasonable results. Thank you again for your time!

MartyG-RealSense commented 2 years ago

The IMU data is noisy by nature so you have not done something wrong.

The rs-motion example program provides the ability for custom adjustment by increasing or decreasing an 'alpha' value to weight the calculations towards the gyro (less sensitivity but more prone to drift) or towards the accelerometer (more sensitivity).

https://github.com/IntelRealSense/librealsense/blob/master/examples/motion/rs-motion.cpp#L117

Cpeck71 commented 2 years ago

Hi Marty, I realize it's been some time since i commented. I think one thing I've found is that when running the rs-enumerate-devices with -c and getting the extrinsic parameters of the devices, the reported translations are "backwards". Meaning when i'm looking at "camera to depth", the translation is actually from depth to camera. I think this is the only way I can resolve the translation vectors as they are in the figure i originally posted.

When it comes to the IMU data, one thing that makes it difficult is the way the accel outputs it's data. It provides the "force acting on it" (i think ive read that in other threads) rather than the instantaneous acceleration, which means that knowledge of the absolute orientation (at least with respect to the direction of gravity) is paramount. I don't blame y'all for that or anything, just an observation. I've 'turned off' using the IMU for the moment to focus on just the image/depth data processing and I'm still having errors with that, so I can't completely fault the IMU.

I'll close this issue for now, but if I get a eureka moment of any major bug I've discovered (on my part or y'alls, but it'll probably be me) I'll re-open just to report it. Thanks so much for y'alls time and effort.

MartyG-RealSense commented 2 years ago

Thanks very much @Cpeck71 - please do feel free to re-open at a future date if you have new insights to share. Good luck!