DriveAGI

This is "The One" project that OpenDriveLab is committed to contribute to the community, providing some thought and general picture of how to embrace foundation models into autonomous driving.

Table of Contents

• NEWS
• At A Glance
• 🚀 Vista (NeurIPS 2024)
• ⭐ GenAD: OpenDV Dataset (CVPR 2024 Hightlight)
• ⭐ DriveLM (ECCV 2024 Oral)
• DriveData Survey
• OpenScene
• OpenLane-V2 Update

NEWS

At A Glance

Vista

Simulated futures in a wide range of driving scenarios by Vista. Best viewed on demo page.

🌏 A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024)

Quick facts:

• Introducing the world's first generalizable driving world model.
• Task: High-fidelity, action-conditioned, and long-horizon future prediction for driving scenes in the wild.
• Dataset: OpenDV-YouTube, nuScenes
• Code and model: https://github.com/OpenDriveLab/Vista
• Video Demo: https://vista-demo.github.io
• Related work: Vista, GenAD

@inproceedings{gao2024vista,
 title={Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability}, 
 author={Shenyuan Gao and Jiazhi Yang and Li Chen and Kashyap Chitta and Yihang Qiu and Andreas Geiger and Jun Zhang and Hongyang Li},
 booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
 year={2024}
}

@inproceedings{yang2024genad,
  title={{Generalized Predictive Model for Autonomous Driving}},
  author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

GenAD: OpenDV Dataset

Examples of real-world driving scenarios in the OpenDV dataset, including urban, highway, rural scenes, etc.

⭐ Generalized Predictive Model for Autonomous Driving (CVPR 2024, Highlight)

Paper | Video | Poster | Slides

🎦 The Largest Driving Video dataset to date, containing more than 1700 hours of real-world driving videos and being 300 times larger than the widely used nuScenes dataset.

• Complete video list (under YouTube license): OpenDV Videos.
  • The downloaded raw videos (mostly 1080P) consume about 3 TB storage space. However, these hour-long videos cannot be directly applied for model training as they are extremely memory consuming.
  • Therefore, we preprocess them into conseductive images which are more flexible and efficient to load during training. Processed images consumes about 24 TB storage space in total.
  • It's recommended to set up your experiments on a small subset, say 1/20 of the whole dataset. An official mini subset is also provided and you can refer to OpenDV-mini for details. After stablizing the training, you can then apply your method on the whole dataset and hope for the best 🤞.
• [ New❗️] Mini subset: OpenDV-mini.
  • A mini version of OpenDV-YouTube. The raw videos consume about 44 GB of storage space and the processed images will consume about 390 GB of storage space.
• Step-by-step instruction for data preparation: OpenDV-YouTube.
• Language annotation for OpenDV-YouTube: OpenDV-YouTube-Language.

Quick facts:

• Task: large-scale video prediction for driving scenes.
• Data source: YouTube, with careful collection and filtering process.
• Diversity Highlights: 1700 hours of driving videos, covering more than 244 cities in 40 countries.
• Related work: GenAD Accepted at CVPR 2024, Highlight
• Note: Annotations for other public datasets in OpenDV-2K will not be released since we randomly sampled a subset of them in training, which are incomplete and hard to trace back to their origins (i.e., file name). Nevertheless, it's easy to reproduce the collection and annotation process on your own following our paper.

@inproceedings{yang2024genad,
  title={Generalized Predictive Model for Autonomous Driving},
  author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

DriveLM

Introducing the First benchmark on Language Prompt for Driving.

Quick facts:

• Task: given the language prompts as input, predict the trajectory in the scene
• Origin dataset: nuScenes, CARLA (To be released)
• Repo: https://github.com/OpenDriveLab/DriveLM, https://github.com/OpenDriveLab/ELM
• Related work: DriveLM, ELM
• Related challenge: Driving with Language AGC Challenge 2024

DriveData Survey

OpenScene

OpenLane-V2 Update