search

ctu-vras / traversability_estimation

Semantic Segmentation of Images and Point Clouds for Traversability Estimation
BSD 3-Clause "New" or "Revised" License
36 stars 6 forks source link
deep-learning image-segmentation mobile-robotics point-cloud-segmentation semantic-segmentation traversability-mapping

readme

Traversability Estimation

Segmentation of Images and Point Clouds for Traversability Estimation.

The module assigns to individual measured LiDAR points either a binary flag or non-negative cost of passing the point by a given robot. Exit from this module could be used as one of the inputs of the planning module. Its use allows planning safe paths around obstacles in a mostly static environment. Example of the module being used in navigation pipeline: video, in simulator.

Installation

Please, follow the instructions in ./docs/install.md.

Images Semantic Segmentation Node

The node takes as input RGB images and intrinsics (from several cameras as well is supported). Output is a set of semantic labels for each input RGB image.

Topics:

Parameters:

Look at segmentation_inferece for more details.

Point Cloud Semantic Segmentation Node

The node takes as input a point cloud from which a depth image is calculated. The depth image is an input for semantic segmentation network (2D-convolutions based) from torchvision.models.segmentation. The network predicts semantic label for each pixel in the depth image. The labels are futher used to output segmented point cloud.

Topics:

Parameters:

Look at cloud_segmentation for more details.

Data

Bag-file sequences are available at: http://subtdata.felk.cvut.cz/robingas/data/. Please, refer to the monoforce package for more details.

Models Training

The following scripts should be run from the ./scripts/tools/ folder:

roscd traversability_estimation/scripts/tools/

Train point cloud segmentation model to predict traversability labels on SemanticKITTI and SemanticUSL data:

python train_depth --datasets SemanticKITTI SemanticUSL --batch_size 4 --output traversability

Train image segmentation model on Rellis3D data:

python train_img --dataset Rellis3DImages --batch_size 4 --architecture fcn_resnet50

Models Evaluation

Evaluate (get IoU score) a point cloud segmentation model trained on TraversabilityClouds data:

python eval_depth --dataset TraversabilityClouds --weights /path/to/deeplabv3_resnet101_lr_0.0001_bs_8_epoch_90_TraversabilityClouds_depth_labels_traversability_iou_0.972.pth --output traversability

3D-Point Cloud Semantic Segmentation Node

The node takes as input a point cloud and utilizes a 3D-convolutions-based network to predict semantic labels for each point. The model is trained using the semi-supervised technique for domain adaptation proposed in T-Concord3D project. Rellis3DClouds data was utilized as the source dataset, while TraversabilityDataset was utilized as the target dataset (using pseudo-labels). Please, have a look at the T-Concord3D project repository for more details about the training procedure.

Topics:

Parameters:

Look at cloud_segmentation_tconcord3d for more details.

Geometric Traversability Node

Manually designed geometric features describing the local neighborhood of points based on:

For more information, please, refer to traversability node implemented in the naex package.

Fused Traversability Node

Method which combines geometric and semantic traversability results. Definitely passable and definitely impassable costs assigned to points values were defined on the basis of geometrical traversability. In the rest of the area (part of point cloud), especially in vegetation where geometrical approach cannot be applied, a model learned from the data was used (semantic traversability estimation).

Topics:

Parameters:

Look at traversability_fusion for more details.

Demos

Citation

Please, cite the papers if you find the package relevant to your research:

@ARTICLE{9699042,
  author={Agishev, Ruslan and Petříček, Tomáš and Zimmermann, Karel},
  journal={IEEE Robotics and Automation Letters},
  title={Trajectory Optimization Using Learned Robot-Terrain Interaction Model in Exploration of Large Subterranean Environments},
  year={2022},
  volume={7},
  number={2},
  pages={3365-3371},
  doi={10.1109/LRA.2022.3147332}
}
@article{agishev2023monoforce,
    title={MonoForce: Self-supervised Learning of Physics-aware Model for Predicting Robot-terrain Interaction},
    author={Ruslan Agishev and Karel Zimmermann and Vladimír Kubelka and Martin Pecka and Tomáš Svoboda},
    year={2023},
    eprint={2309.09007},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}