search

orbbec / OrbbecSDK_ROS2

OrbbecSDK ROS2 wrapper
https://orbbec.github.io/OrbbecSDK_ROS2/
Apache License 2.0
57 stars 19 forks source link

Overwrite of depth image when aligning depth images to color using femto bolt ROS2 Humble #35

Closed peterfryd closed 3 weeks ago

peterfryd commented 1 month ago

I am trying to align my depth images to the color frame while keeping the original unaligned depth image.

I am using ROS2 Humble and a femto-bolt camera with its respective launch file with the parameters depth_registration to 'true' and align_mode to 'SW'. With these parameters, the images from the /camera/depth/image_raw topic are aligned to the /camera/color/image_raw topic which is very good (see SW-picture) , however, the original unaligned depth image is overwritten (on the depth/image_raw topic) in the process which loses a lot of data since the FOV of the depth camera is wider than that of the color camera. If i try to use align_mode 'HW' i get the data of the depth image with its original data untouched but without a matching color image (see HW-picture). In my humble opinion it would make sense to have the aligned data on a separate "aligned" topic so to not overwrite the "raw" topics.

Here are my depth-and color-images with the align_mode HW and SW respectively. The topic data for each image is shown on the right side.

The HW-picture: HW

The SW-picture: SW

Is there a way where i can get acces to both the depth image with the larger FOV (the Depth image from the HW-picture) and the two aligned images (those seen on the SW-picture) on seperate topics?

NathanOrb commented 1 month ago

Femto Bolt doesn't support D2C in HW mode, only SW support. To align the depth and color, we need to either extend or crop the streams to match the different FOV.

peterfryd commented 1 month ago

Arh thanks, that makes sense! Is there a way where I, with the parameters from the launch file, can choose to alter the smaller FOV color stream to match the depth stream instead of it automatically doing it the other way around with the D2C SW mode? Br Peter

NathanOrb commented 1 month ago

Yes, you can set the color and depth resolution for different FOV in the camera launch file.

peterfryd commented 3 weeks ago

I might not have explained it well enough, or i might not understand your solution. I want to have access to the depth image with the full 120 by 120 degree FOV AND a set of aligned images with both RGB and depth data using the 80 by 51 degree FOV. I cant seem to find a way to keep the original 120 by 120 degree FOV depth data and also aligning images since the 120 by 120 FOV depth image is overwritten in order to obtain the set of aligned images by the alignment method. The original depth image therefore ends up using the RGB FOV of 80 by 51 degrees instead, which makes me lose some valuable depth information being used for a different purpose.

I would therefore optimally need 3 pictures being streamed: one 120 by 120 depth image, and a set of aligned depth- and color-image in the RGB FOV (80 by 51).

Alternatively i could accept finding a way to align the RGB image to the depth image using the 120 by 120 FOV and just cropping everything off the RGB image not being registered. Sorry if my prior explanation lead to confusion. I really appreciate the help!

zhonghong322 commented 3 weeks ago

I understand your needs. You want to retain the original Depth image with the original FOV while also having a Depth image aligned with the Color image. In the Orbbec SDK's C++ SDK, this can be done. First, disable D2C to get the original Depth image, then externally call the D2C interface provided by the utility class (ob::CoordinateTransformHelper::transformationDepthFrameToColorCamera) to get the aligned Depth image. However, this functionality is not currently encapsulated in ROS, but we are considering adding it in future ROS releases. Additionally, we will also add C2D functionality in the future.

peterfryd commented 3 weeks ago

Thank you very much for your elaborate answer!