search

introlab / rtabmap_ros

RTAB-Map's ROS package.
http://wiki.ros.org/rtabmap_ros
BSD 3-Clause "New" or "Revised" License
957 stars 557 forks source link

How can I make certain color on the ground as an obstacle? #317

Open ayush-j9 opened 5 years ago

ayush-j9 commented 5 years ago

I want to use this package to create a 2D Occupancy grid map of my house's path way. In addition to the 3D objects, I want to classify the green grass patch as an obstacle too. Is there anything that can help me achieve that? Also, I want to use a Realsense D435 RGBD camera for my purpose.

matlabbe commented 5 years ago

The current approaches embedded in rtabmap are geometric-based segmentation based on normals of the point cloud or if points are over a fixed threshold (e.g., 2D space where we know that everything over 5 cm are obstacles).

To use criteria based on the RGB image (e.g., like semantic segmentation), it is not currently possible. I think the easiest way would be to input the segmented image to rtabmap, telling which pixels should be an obstacle or ground. This way, you could have a floodfill approach or a CNN approach to set grass as obstacle.

The related function to create local oocupancy grid is here, called from here when a new node is created.

cheers, Mathieu

ayush-j9 commented 5 years ago

Thanks for the suggestion.

I am planning to pass in the segmented image to rtabmap. I want to use the 3D map of the environment which contains the point cloud and their RGB data. As far as I can see, it is saved in the mapData. Unfortunately, it seems that the mapData only stores data at a particular pose. Isn't there a topic that contains the entire 3D map which gets updated after loop closures?

I want use that 3D data and then project it on the ground for creating and occupancy grid. I felt that making changes directly in the package will be a very difficult task and so I want to do this separately.

matlabbe commented 5 years ago

I think /rtabmap/cloud_map (PointCloud2 type) is the topic you are looking for.

cheers, Mathieu

ayush-j9 commented 5 years ago

Thanks man. Strangely the /rtabmap/cloud_map did not generate an entire point cloud of the map for the demo_mapping.bag but it did with the stereo_outdoorA.bag

Also, if I pass the semantically segmented RGB images to rtabmap, will it be able to do loop closures since the entire RGB image is now classified with segments and the features could be lost.

matlabbe commented 5 years ago

The cloud_map should represent the whole map, unless memory management is enabled.

For the segmentation of the image, an idea would be to convert the image to grayscale, then convert back in RGB using segmentation colors as tone on the grayscale image. That way, we keep visual features (done on image converted back to grayscale image) and when the cloud is generated, the obstacles could be determined by their color.