How to align depth and color images from a ORIG.BAG file

[FelipeZerokun]

Hello! I am having troubles aligning the color and depth images. I have a Rosbag with data from a RealSense Depth Camera D435i (and other sensors). I only have the rosbag and at the moment I do not have the camera Here is the hardware info:

---

Required Info
Camera Model	D435i
Firmware Version	-
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	6.8.0-48-generic
Platform	PC
SDK Version	-
Language	Python
Segment	Robot

Issue Description

I have read about how to align RGB images and Depth images using the RealSense library and pipeline, also from a rosbag. However, when I try to use config.enable_device_from_file(rosbag_path) I get the following error: Failed to create ros reader: Invalid file format, file does not contain topic "/file_version" nor "/FILE_VERSION"

I also read that some Rosbags cannot be used for the enable_device_from_file() method.

I can extract both images from the Rosbags (I also do this manually because I need to get the data from other sensors) and when checking them I can see they are not aligned. I need them align because I need to identify objects in the images and then check the distance of these objects to the robot.

In the image can be seen the difference between both images and parameters I get from the topic: /camera/color/camera_info'.

I am wondering then, how can I align both images to extract depth information from objects in the RGB image? Thanks :)

[MartyG-RealSense]

Hi @FelipeZerokun If your rosbag was recorded in ROS with a RealSense camera and also non-camera devices - the 'other sensors' that you mention the bag contains data from - then the librealsense SDK may be unable to read the bag. This is because librealsense's bag support is only designed for camera data.

A RealSense user created a rosbag parser at https://github.com/IntelRealSense/librealsense/issues/2215 to read bag files using the rosbag API directly. It is in C++ language rather than Python though.

[FelipeZerokun]

@MartyG-RealSense Thanks, I am checking it out now. I saw that both cameras have Intrinsics and Extrinsics parameters. I guess those parameters are different for the RGB and Depth cameras? Because in my ROS files I can only get the '/camera/color/camera_info' topic with K, D, R and P values, I don't find any depth/camera_info

[MartyG-RealSense]

On the D435-type camera models such as D435i, the five distortion coefficients are all set to zero, as advised at https://github.com/IntelRealSense/librealsense/issues/1430

Yes, each stream type has its own unique intrinsic and extrinsic values.

You can list all of the intrinsic and extrinsic values of a camera with the librealsense SDK's 'rs-enumerate-devices' tool by launching it in calibration information mode with the command below.

rs-enumerate-devices -c

[FelipeZerokun]

Ok, so I imagine I need the actual camera to find these values, right? I don't find these values anywhere in the Rosbags. And, once I have these values, I think I can do camera calibration and image alignment without the actual hardware, correct?

[MartyG-RealSense]

Ideally you would read the intrinsics and extrinsics from the actual camera that was used to record that rosbag file.

If you do not have access to the real camera though, my research indicates that in ROS, intriniscs should be stored in a topic called Camerainfo

https://stevengong.co/notes/CameraInfo

https://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html

You should be able to do calibration and alignment without the camera if you have the intrinsics. An example of this is the RealSense SDK's software-device 'virtual camera' interface for performing tasks such as pointcloud generation without a physical camera by providing a list of intrinsics.

https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device

[FelipeZerokun]

Ok, I couldn't find the values mentioned in the links for the stereo camera (It's like only the mono-camera was recorded in the Rosbags)

D = [ 0.0 , 0.0 , 0.0 , 0.0 , 0.0] K= [ 604.867 , 0.0 , 320.659 ] [ 0.0 , 604.944, 232.351 ] [ 0.0 , 0.0 , 1.0 ]

R = [ 1.0, 0.0, 0.0] [ 0.0, 1.0, 0.0] [ 0.0, 0.0, 1.0]

P= [ 604.867 , 0.0 , 320.659, 0.0 ] [ 0.0 , 604.944, 232.351, 0.0 ] [ 0.0 , 0.0 , 1.0 , 0.0 ]

From all this, it means: fx = 604.867 fy = 604.944 cx = 320.659 cy = 232.351
tx = ty = 0

Is it possible that the camera was configured in a different way? Anyway, I will try to understand why I am getting these values from the Rosbags, and not Stereo intrinsics, and then I'll perform manual calibration and alignment.

[MartyG-RealSense]

Each individual camera has unique intrinsics due to the manufacturing process at the factory, so no two cameras will have the same values. The intrinsic values also change depending upon which resolution is selected.