Depth quality and calibration issue in realsense D435I

[naitiknakrani]

---

Required Info
Camera Model	D435i
Firmware Version	5.13.00.50
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.9.253-tegra
Platform	Nvidia AGX Xavier
SDK Version	{ 2.50.0 }
Language
Segment	VSLAM

Issue Description

We are working to generate 3D map using VSLAM. We are using ROS2 wrapper and realsense2_camera package for invoking d435i camera.

There are few things we observed please help us in understanding these problems.

1. while running vslam node, everytime on chip calibration required from realsense-viewer before launching realsense2_camera (ros2-beta branch) package. In case calibration is not done, we got large odom drift and map construction quality degrades significantly. Is this expected result ? Do anyone face similar issue? Is there any way to provide auto calibration everytime while running launch files ?

2. While getting depth image from texture less white wall, it misses many spots and provides black dots in realsense-viewer with emitter.enabled is "off". and as expected while turning on "laser" in realsense-viewer those depth quality improves as shown in figure. But the problem arise when we turn on laser and do the calibration it resets the setting and turn off "laser". So calibration happens but without laser. Hence, realsense2_camera (ros2-beta branch) shows depth image without laser on, depth image is poor in quality and that impacts SLAM. So is there any way to do calibration while keeping emitter.enabled as "laser" and invoking the realsense2_camera launch files with keeping emitter.enables as "laser"?

So, overall while mapping the environment using realsense d435i in texture full scenes it is quite satisfactory, but when white or plain walls comes because of missing depth details we are not able to generate proper map.

Please help us in finding answers.

[MartyG-RealSense]

Hi @naitiknakrani When using the 2.50.0 version of the RealSense Viewer, the On-Chip calibration interface may not display correctly for some users and disable the emitter to Off status when the 'Calibrate' button is clicked.

A method that can correct this issue with the emitter and the On-Chip interface is to use the previous version 2.49.0 of the Viewer instead. On Ubuntu, doing so would involve installing the 2.49.0 version of the RealSense SDK.

If changing SDK versions would be inconvenient for you (especially in regard to using the ros2_beta wrapper which requires 2,50.0) then you could instead try resetting the camera to its default factory-new calibration in the Viewer. Instructions for doing so and saving the results of the reset to the camera hardware so that it does not have to be repeated can be found at https://github.com/IntelRealSense/librealsense/issues/10182#issuecomment-1019854487

---

If you resolve the calibration issue and the emitter is still disabled in ros2_beta, does using the command below enable it?

ros2 param set /camera/camera depth_module.emitter_enabled true

[naitiknakrani]

@MartyG-RealSense Thanks for reply. As per your suggestion we are trying to reset factory setting found as per https://github.com/IntelRealSense/librealsense/issues/10182#issuecomment-1019854487 but restore factory was not applied. Also, write table link was not enabling. Look at the image.

This was the logs we were getting,

So our calibration issue still persist. Calibration broke always when camera was detached.

[MartyG-RealSense]

The depth stream should be enabled by left-clicking on the red icon beside Stereo Module to turn it to blue (On) before opening the Camera Calibration window and then the Write Table button should be selectable.

Repeatedly generating control_transfer returned errors in the log can indicate that there is a problem with communicating with the camera.

[naitiknakrani]

No success. We were able to write the factory reset data. Please note that previously seen data from tables were same as Restore factory data.

we tried again SLAM but unfortunately we are getting drift and lags when moving camera.

Here we moved camera in some random path and then place to its starting location. We could see drift between odom and camera_link

for more reference, here is the pic of 3D reconstruction from RGB and Depth data. we can see depth image is hardly visible. we kept depth_module.emitter_enabled as True.

[Aki1608]

@MartyG-RealSense After doing the on-chip calibration, when we click "apply new", should it change the data in the camera calibration table? When we do that, it is not changing any value in the calibration table. I am referring to this table.

[MartyG-RealSense]

Hi @Aki1608 Yes, clicking Apply New on the On-Chip calibration interface after a calibration has been successfully completed writes the calibration to the 'calibration table' stored inside the camera hardware.

Do not be concerned if the Calibration Data window does not appear to change after an On-Chip calibration.

[naitiknakrani]

@MartyG-RealSense Is there any chance that 'calibration table' stored in hardware after on-chip-calibration, resets after unplugging camera ?

[MartyG-RealSense]

@naitiknakrani The RealSense IMU's tracking may be negatively affected if there are quick movements or turns, as described in Intel's D435i SLAM guide at the link below.

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

To quote the guide:

---

The built-in IMU can only keep track for a very short time. Moving or turning too quickly will break the sequence of successful point cloud matches and will result in the system losing track. It could happen that the system will recover immediately if stopped moving but if not, the longer the time passed since the break, the farther away it will drift from the correct position. The odds for recovery get very slim, very quickly.

[MartyG-RealSense]

I would advise only using the Calibration Data window for the purposes of resetting the calibration table to factory-default values and ignoring the intrinsics data shown in the window. If you have used On-Chip to write a calibration with a Good score to the camera hardware then that stored calibration can be trusted as reliable. Resetting the calibration table is primarily useful for correcting corruptions in the calibration table that may be preventing calibration from working correctly.

[naitiknakrani]

@MartyG-RealSense Hi, can we see and check calibration table data after on-chip-calibration process with good score which are stored in internal storage? We wish to understand difference between those data with factory-reset values ? May that help us in further progress.

Yes, we agreed that doing quick and jerky movement of camera while performing SLAM lost its track. We observed that ( Probably "Tracker is lost" logs are due to that ?) and avoid it in our experiment.

[Aki1608]

@MartyG-RealSense on-chip calibration won't affect the IMU right? Isn't there a different method to calibrate the IMU? What we try to figure out here is a way so that we don't have to calibrate the camera every time we want to use it. Somehow we want the camera to restore the last calibrated data whenever we turn it on.

[MartyG-RealSense]

@Aki1608 You are correct. On-Chip calibration does not affect the IMU. Intel provide a separate Python-based tool at the link below for IMU calibration.

https://github.com/IntelRealSense/librealsense/tree/master/tools/rs-imu-calibration

A RealSense camera should not require a re-calibration every time that it is used if the calibration is saved to the EEPROM flash memory storage space inside the camera hardware (located on its depth module board) after it has been performed, as the camera should use its internally stored calibration each time that it is used.

@naitiknakrani The On-Chip calibration tool supports a RealSense SDK programming instruction called rs2_get_raw_data for saving the calibration to disk.

https://intelrealsense.github.io/librealsense/doxygen/rs_8h.html#a0c01013907d7f47d2a85712e7ca34a7f

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-b-details-on-self-calibration-flow-and-c-api

[naitiknakrani]

@MartyG-RealSense We found lots of code samples from https://dev.intelrealsense.com/docs/code-samples

We need "rs-on-chip-calibration" similar code that performs on-chip-calibration from command prompt inside the Docker (Isaac_ros_common). Can any intel developers help us on that?

[MartyG-RealSense]

The concept of creating an interface for accessing On-Chip calibration from within ROS was previously suggested by a RealSense ROS user in a 'pull' that they submitted for the wrapper at https://github.com/IntelRealSense/realsense-ros/pull/1256

In the discussion on that pull page, the developer of the RealSense ROS1 wrapper preferred controlling On-Chip via librealsense by using Python code such as the example in Intel's calibration white-paper document at the link below.

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-c-on-chip-calibration-python-api

If you need to be able to control On-Chip calibration from ROS then you could look at creating and launching from the ROS terminal a Python node script like the wrapper examples here:

https://github.com/IntelRealSense/realsense-ros/tree/development/realsense2_camera/scripts

[naitiknakrani]

@MartyG-RealSense We are trying to use pyrealsense2 api. From your previous comment https://github.com/IntelRealSense/librealsense/issues/6964#issuecomment-665650908 we understood that librealsense has to build from source.

we can also see installation guide for python wrapper https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python. It shows installation upto ubuntu 14.04 and 16.04. Would that also work on 18.04 or any specific branch will work for 18.04 ?

Any link showing clear step for installing librealsense and pyrealsense2 from source on Jetson AGX?

[MartyG-RealSense]

There is a guide at https://github.com/IntelRealSense/librealsense/issues/6964#issuecomment-707501049 for installing librealsense and pyrealsense2 together from source code on Jetson.

The pyrealsense2 wrapper works with Ubuntu 18.04 too, yes.

[MartyG-RealSense]

Hi @Aki1608 and @naitiknakrani Do you require further assistance with this case, please? Thanks!

[Aki1608]

Hi @MartyG-RealSense. We did find many useful informations. Thank you for your help. 😊

[MartyG-RealSense]

Thanks very much @Aki1608 for your update!

[MartyG-RealSense]

Hi @naitiknakrani Do you have an update about this case that you can provide, please? Thanks!

[naitiknakrani]

Hi @MartyG-RealSense.

Like you said, in SDK version 2.50.0, on-chip-calibration step makes the emitter off. There is no option other than using it without Emitter enabled as we are working on ROS2 Humble docker.

We are working on methods to perform on-chip-calibration using python code inside docker. We need pyrealsense2 package inside docker. We will update you on the process once we got some success.

[MartyG-RealSense]

@naitiknakrani When Intel released SDK 2.50.0 they also released a Docker tutorial guide that includes support for Python bindings in the Dockerfile description. Is this guide of any help to your investigation of pyrealsense2 within Docker?

https://github.com/IntelRealSense/librealsense/tree/master/scripts/Docker

[MartyG-RealSense]

Hi @naitiknakrani Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[naitiknakrani]

Hi @naitiknakrani When using the 2.50.0 version of the RealSense Viewer, the On-Chip calibration interface may not display correctly for some users and disable the emitter to Off status when the 'Calibrate' button is clicked.

@MartyG-RealSense What is relation between on-chip calibration and laser emitter status for the latest 2.51.1 release? Is there any update on this?

[MartyG-RealSense]

Hi @naitiknakrani I tested with the 2.51.1 version of the RealSense Viewer and the IR emitter issue that is present in 2.50.0 is still in 2.51.1. The 2.49.0 Viewer's on-chip calibration interface continues to work correctly.

[naitiknakrani]

Thanks @MartyG-RealSense for the kind reply and update.