Pixel Level Segmentation Based Drivable Road Region Detection and Steering Angle Estimation Method for Autonomous Driving on Unstructured Roads

Abstract

With the recent emergence of deep learning, computer vision-based applications have demonstrated better applicability in accomplishing driving tasks including drivable road region detection, lane keeping and steering control in self-driving cars. Till recently, numerous lane-marking detection based steering control and lane keeping methods have been proposed to perform autonomous driving on urban well-structured roads. But the matter of fact is that these methods are not feasible on roads where lane markings are not available or faded over time which makes drivable road region detection a crucial task. Moreover, it is highly arduous task to estimate steering angle on deteriorated roads using existing road detection and steering angle estimation methods. To the best of our knowledge, there is no standard benchmark available for drivable road region detection and steering angle estimation on unstructured roads. To this end, we present a large-scale dataset for drivable region road detection, comprising of 15,000-pixel level high quality fine annotations. Alongside dataset, we also present an end-to-end drivable road region detection and steering angle estimation method to ensure the autonomous driving on generalized urban, rural, and unstructured road conditions. The proposed method performs pixel-level segmentation to extract drivable road region and quantifies lane interception to estimate the steering angle of self-driving cars. A comprehensive qualitative and quantitative analysis has been carried out to demonstrate the effectiveness of our proposed dataset, road detection and steering angle estimation methods.

End to End semantic segmentation based drivable road detection and steering angle estimation on unstructured roads for self driving cars

Discription of Project

A PyTorch implementation of drivable road region detection and steering angle estimation methods proposed for the autonomous vehicle. The dataset has been taken from CARL-DATASET.

Prerequisites

1. Windows
2. Anaconda Python
3. PyTorch (https://pytorch.org/get-started/locally/)
4. Albumentations (https://pypi.org/project/albumentations/), (pip install albumentations)
5. Tensorboard (https://pypi.org/project/tensorboard/), (pip install tensorboard)
6. TensorboardX (https://pypi.org/project/tensorboardX/), (pip install tensorboardX)

   Dataset

   We have extended CARL-Dataset for road detection and segmentation task. As CARL-Dataset has been constructed over video sequences from 100+ cities of Pakistan.

This dataset contains diversities in terms of road types such as

• Highways
• Motorways
• Rural and Urban streets
• Provincial
• District
• Hilly and Distressed roads

To ensure generalization of our proposed method, equal subsets of images from video sequences of all types of captured roads have been selected for the training and evaluation of proposed method.

Some samples taken from the video sequences of diverse roadways are shown Below.

Pixel-Level Annotations of dataset

• We present a pixel-level annotations benchmark comprising of high resolution (1920×1080 ) images and annotations, having road variability and diversity.
• Our dataset consists of 15000 pixel-level annotations, in which each image is labeled as (i) road, and (ii) background, respectively.
• To accomplish this task, we have used annotation tool SuperAnnotate to perform binary class-based pixel-level annotations.
• In our binary class annotations, we have used RGB based colored polygons to represent defined classes.

Some of the pixel-level annotation examples are depicted below.

Download the complete dataset from Here.

Description of folders and scripts

My project includes the following files scripts and folders:

• CARL_DATASET folder consists of six subfolders i.e. test, test_label, train, train_label, val, and val_label containing road images and corresponding label images.
• runs folder contains the segmentation logs of the validation data.
• train_seg_maps folder is used to save the prediction of the trained model on rondomly selected image during validation process.
• utils folder contains the files such as (i) helper.py and (ii) metrics.py.
• config.py is the configation file, where dataset path, name and number of classes mentioned. Check every other parameters in config.py as per your requirement.
• dataset.py includes the functions regarding dataset.
• helper.py includes some helper functions.
• train.py is used to train the model.
• road_detection_test.py is used to test the model on randomly selected image.
• road_detection_angle_estimation_test.py is used to test the model on the test image and predict the angle.
• SteeringAngleUtils.py is used for computing the steering angle against the given test image.
• test_vid.py is used to detect driveable road region on given video input.
• model.py is used to train the model from scratch.
• engine.py is used to save the model as a model.pth.
• metrics.py includes helper functions in order to measure the performance matrics.

How to run the Python scripts

For Training

• Train the model for the first time on the road detection dataset CARL-DATASET. Before starting training, please place the downloaded dataset folders (train, train_labels, test, test_labels, val, and val_labels) in CARL-Dataset directory. After completion, verify the root paths and other configurations in 'config.py'.

  python train.py --resume-training no

  For Testing

  Test the model on the image

• Use this python script to apply pixel level segmentation on any image of your choice.

  python test_road_detection.py --model-path <path to saved checkpoint/weight file> --input <path to vid>.

  example: python test.py --model-path model.pth --input abc.jpg

  Test the model on the video

• Use this python script to apply pixel level segmentation on any videos of your choice.

  python test_vid.py --input <path to vid> --model-path <path to saved checkpoint/weight file>.

  example: python test_vid.py --input DSC_0006.mp4 --model-path model.pth

  Result

  Prediction results of the model over images

  

Prediction results of the model over video

https://user-images.githubusercontent.com/71174927/147828349-4829a38e-8310-4bab-be8e-b1cc58a079e3.mp4

https://user-images.githubusercontent.com/71174927/147775570-bc58cbf2-7bcb-4c8b-9c40-09ffe39feb7f.mp4

Citation

• If you follow the code or ideas from our paper, please cite our paper:

  @ARTICLE{9646953,
  author={Rasib, Marya and Butt, Muhammad Atif and Riaz, Faisal and Sulaiman, Adel and Akram, Muhammad},
  journal={IEEE Access}, 
  title={Pixel Level Segmentation Based Drivable Road Region Detection and Steering Angle Estimation Method for Autonomous Driving on Unstructured Roads}, 
  year={2021},
  volume={9},
  number={},
  pages={167855-167867},
  doi={10.1109/ACCESS.2021.3134889}}

Acknowledgments

This code inspired by Sovit Ranjan Rath. Some snippets of steering angle estimation code from David Tain.