search

PRBonn / bonnet

Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics.
GNU General Public License v3.0
325 stars 89 forks source link
convolutional-neural-networks robotics ros semantic-segmentation tensorflow tensorrt

readme

Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics.

Build Status

By Andres Milioto @ University of Bonn.

(for the new Pytorch version, go here)

Image of cityscapes Cityscapes Urban Scene understanding.

Image of Persons Person Segmentation

Image of cwc Crop vs. Weed Semantic Segmentation.

Description

This code provides a framework to easily add architectures and datasets, in order to train and deploy CNNs for a robot. It contains a full training pipeline in python using Tensorflow and OpenCV, and it also some C++ apps to deploy a frozen protobuf in ROS and standalone. The C++ library is made in a way which allows to add other backends (such as TensorRT and MvNCS), but only Tensorflow and TensorRT are implemented for now. For now, we will keep it this way because we are mostly interested in deployment for the Jetson and Drive platforms, but if you have a specific need, we accept pull requests!

The networks included is based of of many other architectures (see below), but not exactly a copy of any of them. As seen in the videos, they run very fast in both GPU and CPU, and they are designed with performance in mind, at the cost of a slight accuracy loss. Feel free to use it as a model to implement your own architecture.

All scripts have been tested on the following configurations:

We also provide a Dockerfile to make it easy to run without worrying about the dependencies, which is based on the official nvidia/cuda image containing cuda9 and cudnn7. In order to build and run this image with support for X11 (to display the results), you can run this in the repo root directory (nvidia-docker should be used instead of vainilla docker):

  $ docker pull tano297/bonnet:cuda9-cudnn7-tf17-trt304
  $ nvidia-docker build -t bonnet .
  $ nvidia-docker run -ti --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v $HOME/.Xauthority:/home/developer/.Xauthority -v /home/$USER/data:/shared --net=host --pid=host --ipc=host bonnet /bin/bash

-v /home/$USER/data:/share can be replaced to point to wherever you store the data and trained models, in order to include the data inside the container for inference/training.

Deployment

Training

Pre-trained models

These are some models trained on some sample datasets that you can use with the trainer and deployer, but if you want to take time to write the parsers for another dataset (yaml file with classes and colors + python script to put the data into the standard dataset format) feel free to create a pull request.

If you don't have GPUs and the task is interesting for robots to exploit, I will gladly train it whenever I have some free GPU time in our servers.

License

This software

Bonnet is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Bonnet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Pretrained models

The pretrained models with a specific dataset keep the copyright of such dataset.

Citation

If you use our framework for any academic work, please cite its paper.

@InProceedings{milioto2019icra,
author = {A. Milioto and C. Stachniss},
title = {{Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics using CNNs}},
booktitle = {Proc. of the IEEE Intl. Conf. on Robotics \& Automation (ICRA)},
year = 2019,
codeurl = {https://github.com/Photogrammetry-Robotics-Bonn/bonnet},
videourl = {https://www.youtube.com/watch?v=tfeFHCq6YJs},
}

Our networks are strongly based on the following architectures, so if you use them for any academic work, please give a look at their papers and cite them if you think proper:

Other useful GitHub's:

Contributors

Milioto, Andres

Special thanks to Philipp Lottes for all the work shared during the last year, and to Olga Vysotka and Susanne Wenzel for beta testing the framework :)

Acknowledgements

This work has partly been supported by the German Research Foundation under Germany's Excellence Strategy, EXC-2070 - 390732324 (PhenoRob). We also thank NVIDIA Corporation for providing a Quadro P6000 GPU partially used to develop this framework.

TODOs