PreSight
[ECCV 2024] PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors
[
Introduction
This repository is an official implementation of PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors.
News
-
[2024/07/20]: :tada: We have released the code of PreSight!
-
[2024/07/09]: :confetti_ball: Our paper has been accepted by the The 18th European Conference on Computer Vision (ECCV 2024)! Our code will be release this month. Stay tuned!
Main Results
Online HD Mapping
| Model | Metric | w. Prior | Ped Crossing | Divider | Boundary | All | Runtime (FPS) |
|---|---|---|---|---|---|---|---|
| StreamMapNet | AP | × | 10.19 | 11.26 | 11.87 | 11.10 | 22.4 |
| StreamMapNet | AP | ✓ | 21.11 | 23.73 | 32.31 | 25.72 (+14.62) | 21.9 |
| MapTR | AP | × | 4.97 | 8.20 | 9.83 | 7.67 | 25.2 |
| MapTR | AP | ✓ | 16.18 | 19.04 | 34.14 | 23.12 (+15.45) | 23.2 |
| BEVFormer | IoU | × | 14.90 | 29.88 | 32.74 | 25.84 | 15.5 |
| BEVFormer | IoU | ✓ | 16.37 | 34.82 | 51.66 | 34.28 (+8.44) | 14.3 |
Occupancy
| Method | w. Priors | mIoU | Dynamic | Static | others | barrier | bicycle | bus | car | constr. vehicle | motorcycle | pedestrian | traffic cone | truck | drive surface | other flat | sidewalk | terrain | manmade | vegetation | Runtime (FPS) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BEVDet | × | 29.3 | 24.4 | 38.2 | 1.5 | 42.4 | 11.0 | 43.0 | 47.1 | 19.1 | 23.3 | 23.4 | 19.5 | 37.8 | 72.9 | 11.6 | 30.9 | 48.6 | 32.7 | 32.5 | 5.1 |
| BEVDet | ✓ | 33.7 (+4.4) | 24.4 | 50.5 (+12.3) | 1.2 | 40.1 | 14.8 | 42.1 | 48.3 | 15.7 | 26.4 | 24.4 | 18.7 | 37.2 | 81.8 | 15.2 | 40.3 | 60.5 | 50.4 | 54.9 | 4.9 |
| FB-Occ | × | 30.0 | 25.1 | 39.2 | 9.2 | 37.2 | 21.8 | 41.6 | 43.4 | 15.8 | 27.3 | 25.4 | 23.8 | 30.3 | 74.7 | 17.3 | 33.0 | 50.6 | 28.2 | 31.1 | 9.1 |
| FB-Occ | ✓ | 34.3 (+4.3) | 25.4 | 50.7 (+11.5) | 9.3 | 38.3 | 21.0 | 40.3 | 45.0 | 15.9 | 29.9 | 26.0 | 23.8 | 30.2 | 82.3 | 18.5 | 39.1 | 61.2 | 48.0 | 54.7 | 8.6 |
Getting Started
To get started, please follow the instructions below step-by-step.
Pretrained Weights
Extracted Priors
| Boston-Seaport | Singapore-Onenorth | Singapore-Queenstown | Singapore-Hollandvillage | |
|---|---|---|---|---|
| Google Drive | Download | Download | Download | Download |
Perception Models
| Vectorized Online Mapping | Occupancy Prediction | |
|---|---|---|
| Google Drive | Download | Download |
TODO
- [ ] Add scripts to inference per-image monocular depth using Depth-Anything to enable training NeRFs with monocular-depth loss. Monocular-depth loss improves visualization quality but do not improve downstream perception metrics.
Acknowledgement
This project builds upon the outstanding work of several open-source projects. We extend our sincere thanks to the following codebases:
Citation
If you find our work useful in your research, please consider citing:
@article{yuan2024presight,
title={PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors},
author={Yuan, Tianyuan and Mao, Yucheng and Yang, Jiawei and Liu, Yicheng and Wang, Yue and Zhao, Hang},
journal={arXiv preprint arXiv:2403.09079},
year={2024}
}