Segmentation of the Driving Environment

[xXBasti]

Objective: Segmentation of the Camera Image

Segmentation is a useful tool to perceive the environment and extracts a very rich representation. In combination with Depth Perception, this can lead to a very accurate snapshot of the track.

Classes:

• Line
• Street
  • Drivable
  • Non-Drivable
    • Barred Area
    • Parking Spot
    • ...
  • Road Marking
    • Turning Signal
    • Speed Limit
    • ...
• Obstacle
  • Moving
  • Static
• Pedestrian
• Traffic sign
• Other (surrounding track etc.)

Network architecture

• FCN-based
  • Pros: simple, fast, many implementations exist, pretrained weights available, easy to train
  • Cons: not state-of-the-art anymore
• DeepLab-based
  • Pros: very good performance (still state-of-the-art, implementations in TF exist, pretrained available)
  • Cons: harder to train, relatively performance-hungry (but fast version with MobileNet V2 backend exists)
• SCNN-based
  • Pros: extremely fast, even on slow hardware
  • Cons: performance might not be ideal

Data sources:

• Use real-world datasets like Berkeley Deep Drive, KITTIor CityScapes (however, these might not directly be transferable to our domain -> Domain Adaptation)
• Use our environment generator (originally for Gazebo). A possibility is to render the track in 3D software like Blender to create a more realistic image. Segmentation ground truth can be generated by an additional render pass.

Constraints:

• The network has to run at least at 10 FPS on our on-board hardware (A Jetson TX2 )
• This also includes communication with the ROS core running on the main Intel NUC board

Benchmarks

• IoU and other scores on common semantic segmentation datasets
• Empiric evaluation in the real world

Tasks:

• [ ] Checkout state of the art segmentation algorithms for mobile robots.
• [ ] Define useful classes for the segmentation and compare it with different datasets (CityScapes, BDD, Kitti).
• [ ] Train a model with a public dataset or our data and test performance on the Jetson
• [ ] Fine tune the model on our data

[MaslinuPoimal]

Alternative to separate lane and object detection algorithms, but probably much more performance-intensive.