search

introlab / rtabmap_ros

RTAB-Map's ROS package.
http://wiki.ros.org/rtabmap_ros
BSD 3-Clause "New" or "Revised" License
986 stars 555 forks source link

How to make RTAB-Map contains semantic information? #667

Open YvonneZhao0305 opened 2 years ago

YvonneZhao0305 commented 2 years ago

Hello, I want to input the results of instance segmentation and geometric segmentation (pointclouds with semantic information) into rtabmap, but I find that the input of rtabmap is RGB image and depth image. If the pointclouds with semantic information is projected onto a two-dimensional plane and then transferred to rtabmap, it may lose semantic information (not tried yet). What is a good way to transfer semantic information to rtabmap? @matlabbe Thanks.

matlabbe commented 2 years ago

You can send the image with semantic color instead of the real RGB image, however, visual loop closure detection won't work. If you have already point cloud with semantic information, a hack would be to feed the point cloud (subscribe_scan_cloud=true). That cloud could be XYZRGB format or XYZI format. The semantic may be encoded in the "RGB" or "I" part of the point cloud.

YvonneZhao0305 commented 2 years ago

You can send the image with semantic color instead of the real RGB image, however, visual loop closure detection won't work. If you have already point cloud with semantic information, a hack would be to feed the point cloud (subscribe_scan_cloud=true). That cloud could be XYZRGB format or XYZI format. The semantic may be encoded in the "RGB" or "I" part of the point cloud.

Using voxblox++ for reference, I combine semantic segmentation and geometric segmentation. For each frame of RGB and depth image input, the segments (point cloud with label and specific color) of the current frame will be obtained, and the smaller segments will be filtered out.

  1. In the first scheme, segments with semantic information are projected to rgb_image to obtain proj_image，RTAB-Map utilizes proj_image and depth_image for localization and mapping.
  2. The second scheme takes the segments of each frame as scan_cloud, but segments is not the complete point cloud information of the current frame.

Since the segmentation module will take a long time, I expect that the running speed of the system will slow down when combined with RTAB-Map, and the input of low frame rate (point cloud or rgbd image) will also affect the localization effect. Is there a better solution?

matlabbe commented 2 years ago

It depends on how slow the system get. RTAB-Map updates by default at 1 Hz, so if your segmentation can be done under 1 second, there would be not really a slow down for localization.

YvonneZhao0305 commented 2 years ago

You can send the image with semantic color instead of the real RGB image, however, visual loop closure detection won't work. If you have already point cloud with semantic information, a hack would be to feed the point cloud (subscribe_scan_cloud=true). That cloud could be XYZRGB format or XYZI format. The semantic may be encoded in the "RGB" or "I" part of the point cloud.

Hello, is there a way to input semantic images and use loop detection? I want to use loop detection to merge multiple sessions of the semantic map. I hope I can get your help. I've been stuck here for a long time. THANKS @matlabbe