ADFuzz

Introduction

An open-source software package for fuzzing autonomous driving systems in high-fidelity simulators. It is also currently actively maintained and developed.

Requirements

OS: Ubuntu 18.04, 20.04
CPU: >= 8 cores
GPU: >= 6GB memory (>= 8GB if the perception module of Apollo is used)

Current support of stacks

LBC + CARLA 0.9.9
Apollo(6.0 or later) + SVL 2021.3
No Simulation (Dataset)
No Simulation (Function)

Current support of algorithms and variations

ADFuzz currently support several algorithms and variations listed below. The relevant algorithm_name and key parameters are also mentioned.

Algorithms

NSGA2-SM (-a nsga2 --rank_mode regression_nn --use_single_objective 0 --only_run_unique_cases 0 --regression_nn_use_running_data 0 --warm_up_path <path-to-warm-up-folder> --warm_up_len 500)
NSGA2-DT (-a nsga2-dt --use_single_objective 0 --only_run_unique_cases 0 --outer_iterations 3 --n_gen 5)
AV-Fuzzer (-a avfuzzer --only_run_unique_cases 0)
AutoFuzz (GA-UN-NN-GRAD) (-a nsga2-un --rank_mode adv_nn )

Baselines/Variations

Random (-a random --only_run_unique_cases 0)
Random-UN (-a random-un)
GA (-a nsga2 --only_run_unique_cases 0)
GA-UN (-a nsga2-un)
NSGA2-UN-SM-A (-a nsga2 --rank_mode regression_nn --use_single_objective 0)

Additional Explanation

It should be noted that for NSGA2-SM, additional parameters like warm_up_path and warm_up_len must be specified. For AutoFuzz (GA-UN-NN-GRAD) and NSGA2-UN-SM-A, they can also be specified. warm_up_path refers to the result folder of a run of the initial warm-up stage. Algorithms like Random and GA are commonly used. warm_up_len refers to the results of how many simulation instances from this warm-up stage are leveraged.

Found Traffic Violation Demos

pid-1 controller collides with a pedestrian:

pid-2 controller collides with the stopped leading car:

lbc controller is going wrong lane:

Apollo6.0 collides with a school bus:

Preparation

Install pyenv and python3.8

install pyenv

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

install python

PATH=$HOME/.pyenv/bin:$HOME/.pyenv/shims:$PATH
pyenv install -s 3.8.5
pyenv global 3.8.5
pyenv rehash
eval "$(pyenv init -)"

add the following lines to the end of ~/.bashrc to make sure pyenv is active when openning a new terminal

PATH=$HOME/.pyenv/bin:$HOME/.pyenv/shims:$PATH
eval "$(pyenv init -)"

Environment Setup

In ~/Docuements/self-driving-cars,

git clone https://github.com/AIasd/ADFuzz.git

Install environment

pip3 install -r requirements.txt

Install pytorch on its official website via pip.

Install pytroch-lightening

pip3 install pytorch-lightning==0.8.5

Documentations for the setup and instruction for each stack

CARLA0.9.9+LBC

SVL2021.3+Apollo Master

No Simulation (Dataset)

No Simulation (Function)

CARLA0.9.11+OpenPilot

Citing

If you use the project in your work, please consider citing the following works:

@misc{zhong2021neural,
      title={Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles},
      author={Ziyuan Zhong and Gail Kaiser and Baishakhi Ray},
      year={2021},
      eprint={2109.06126},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

and

@misc{https://doi.org/10.48550/arxiv.2109.06404,
  doi = {10.48550/ARXIV.2109.06404},
  url = {https://arxiv.org/abs/2109.06404},
  author = {Zhong, Ziyuan and Hu, Zhisheng and Guo, Shengjian and Zhang, Xinyang and Zhong, Zhenyu and Ray, Baishakhi},
  keywords = {Robotics (cs.RO), Artificial Intelligence (cs.AI), Machine Learning (cs.LG), Software Engineering (cs.SE), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Detecting Multi-Sensor Fusion Errors in Advanced Driver-Assistance Systems},
  publisher = {arXiv},
  year = {2021},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Reference

This repo leverages code from Carla Challenge (with LBC supported) and pymoo

AIasd / ADFuzz

readme