Point Cloud from multiple .bag files (Python)

[AlexandrineRibeiro]

---

Required Info
Camera Model	D435
Operating System & Version	Win 10
Platform	PC
Language	Python

Issue Description

I have recorded an object using 2 cameras in different positions and saved the .bag files. I want to create the pointclouds of each file and merge them into one. I can do this for each file individually but not for both at the same time. When I read both files simultaneously:

• I can read the frames and display them (depth and color)
• I can't generate both pointclouds: one of them (the second one) appears well and the other one appears all in the same color.

It seems like a problem when setting the pc = rs.pointcloud variable

Is there anyone that can help me?

Many Thanks

[MartyG-RealSense]

Hi @AlexandrineRibeiro The most problematic aspect may be using bag files. The ply point cloud format would be much more suited to "stitching" together pre-recorded data.

It sounds as though you are seeking a mostly automated approach where the captures are taken and then a combined point cloud is generated automatically. Is this correct, please?

If you are using one computer with 2 cameras, I wonder if using ROS to perform a live point cloud capture from different angles and then stitch the clouds might be a viable option for you.

https://github.com/IntelRealSense/realsense-ros/wiki/Showcase-of-using-2-cameras

[AlexandrineRibeiro]

Hi @MartyG-RealSense. My goal was to do it using only python and librealsense library.

If it is possible to generate the poincloud for each file individually, I don't understand why it is not possible to generate for both at the same time.

Everything works correctly but one appears correctly colored and the other appears all of the same color.

[MartyG-RealSense]

If you do not want to use an additional platform such as ROS or Open3D to perform the cloud alignment, the best way to approach it using only Python is likely to be to align the clouds with an affine transform, as described in the link below.

https://community.intel.com/t5/Items-with-no-label/Combining-multiple-depth-streams-into-one/m-p/483016/highlight/true#M5089

An example of an affine transform instruction in librealsense is rs2_transform_point_to_point

https://github.com/IntelRealSense/librealsense/issues/5583

[AlexandrineRibeiro]

Is there any way for me to find out the position of the cameras relative to each other in order to use this affine transformation? Using a target for example?

[MartyG-RealSense]

A target-based way to do this with a target is with aruco image tags that can be applied to the surface of objects. Google for affine transform aruco multiple cameras pose for examples of this method.

An example result from such a search is a paper about calibrating a multi-camera rig with a ChAruco board.

https://trepo.tuni.fi/bitstream/handle/123456789/27414/todorov.pdf

This target method is reminiscent of the official OEM Calibration Target system for 400 Series cameras in the RealSense store that can calibrate multiple cameras.

https://store.intelrealsense.com/buy-intel-realsense-d400-cameras-calibration-target.html

In regard to alternatives to affine transform for point cloud stitching: in a 2018 webinar session, Intel RealSense team members spoke about the ability of a tool called Vicalib to stitch point clouds with RealSense 400 Series cameras. They said "It uses a board that you show to each of the cameras in turn, and it establishes overlapping regions to then minimize the pose of each of those together".

https://github.com/arpg/vicalib

I have heard that Vicalib may have become incompatible with RealSense since then (I do not have confirmation of this though).

[MartyG-RealSense]

Hi @AlexandrineRibeiro Do you require further assistance with this case, please? Thanks!

[AlexandrineRibeiro]

Hi @MartyG-RealSense, I don't need more assistance with this.

Feel free to close this issue.

Alexandrine

[MartyG-RealSense]

Thanks very much for the update @AlexandrineRibeiro