The repository contains the official TensorFlow code used in our papers:
Semantic segmentation benefits robotics related applications especially autonomous driving. Most of the research on semantic segmentation is only on increasing the accuracy of segmentation models with little attention to computationally efficient solutions. The few work conducted in this direction does not provide principled methods to evaluate the different design choices for segmentation. In RTSeg, we address this gap by presenting a real-time semantic segmentation benchmarking framework with a decoupled design for feature extraction and decoding methods. The code and the experimental results are presented on the CityScapes dataset for urban scenes.
Encoder | Skip | U-Net | DilationV1 | DilationV2 |
---|---|---|---|---|
VGG-16 | Yes | Yes | Yes | No |
ResNet-18 | Yes | Yes | Yes | No |
MobileNet | Yes | Yes | Yes | Yes |
ShuffleNet | Yes | Yes | Yes | Yes |
NOTE: The rest of the pretrained weights for all the implemented models will be released soon. Stay in touch for the updates.
Model | GFLOPs | Class IoU | Class iIoU | Category IoU | Category iIoU |
---|---|---|---|---|---|
SegNet | 286.03 | 56.1 | 34.2 | 79.8 | 66.4 |
ENet | 3.83 | 58.3 | 24.4 | 80.4 | 64.0 |
DeepLab | - | 70.4 | 42.6 | 86.4 | 67.7 |
SkipNet-VGG16 | - | 65.3 | 41.7 | 85.7 | 70.1 |
ShuffleSeg | 2.0 | 58.3 | 32.4 | 80.2 | 62.2 |
SkipNet-MobileNet | 6.2 | 61.5 | 35.2 | 82.0 | 63.0 |
Encoder | Decoder | Coarse | mIoU |
---|---|---|---|
MobileNet | SkipNet | No | 61.3 |
ShuffleNet | SkipNet | No | 55.5 |
ResNet-18 | UNet | No | 57.9 |
MobileNet | UNet | No | 61.0 |
ShuffleNet | UNet | No | 57.0 |
MobileNet | Dilation | No | 57.8 |
ShuffleNet | Dilation | No | 53.9 |
MobileNet | SkipNet | Yes | 62.4 |
ShuffleNet | SkipNet | Yes | 59.3 |
GFLOPs is computed on image resolution 360x640. However, the mIOU(s) are computed on the official image resolution required by CityScapes evaluation script 1024x2048.
** Regarding Inference time, issue is reported here. We were not able to outperform the reported inference time from ENet architecture it could be due to discrepencies in the optimization we perform. People are welcome to improve on the optimization method we're using.
The file named run.sh provide a good example for running different architectures. Have a look at this file.
python3 main.py --load_config=[config_file_name].yaml [train/test] [Trainer Class Name] [Model Class Name]
python3 main.py --load_config=unet_resnet18_test.yaml test Train LinkNET
python3 main.py --load_config=unet_resnet18_test.yaml inference Train LinkNET
config/experiments_config/unet_resnet18_test.yaml
and modify it or create another .yaml configuration file depending on your needs.NOTE: The current code does not contain the optimized code for measuring inference time, the final code will be released soon.
Python 3 and above
tensorflow 1.3.0/1.4.0
numpy 1.13.1
tqdm 4.15.0
matplotlib 2.0.2
pillow 4.2.1
PyYAML 3.12
pip install -r [requirements_gpu.txt] or [requirements.txt]
If you find RTSeg useful in your research, please consider citing our work:
@ARTICLE{2018arXiv180302758S,
author = {{Siam}, M. and {Gamal}, M. and {Abdel-Razek}, M. and {Yogamani}, S. and
{Jagersand}, M.},
title = "{RTSeg: Real-time Semantic Segmentation Comparative Study}",
journal = {ArXiv e-prints},
archivePrefix = "arXiv",
eprint = {1803.02758},
primaryClass = "cs.CV",
keywords = {Computer Science - Computer Vision and Pattern Recognition},
year = 2018,
month = mar,
adsurl = {http://adsabs.harvard.edu/abs/2018arXiv180302758S},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
If you find ShuffleSeg useful in your research, please consider citing it as well:
@ARTICLE{2018arXiv180303816G,
author = {{Gamal}, M. and {Siam}, M. and {Abdel-Razek}, M.},
title = "{ShuffleSeg: Real-time Semantic Segmentation Network}",
journal = {ArXiv e-prints},
archivePrefix = "arXiv",
eprint = {1803.03816},
primaryClass = "cs.CV",
keywords = {Computer Science - Computer Vision and Pattern Recognition},
year = 2018,
month = mar,
adsurl = {http://adsabs.harvard.edu/abs/2018arXiv180303816G},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Real-time Motion Segmentation using 2-stream shuffleseg Code