search

OpenDriveLab / OccNet

[ICCV 2023] OccNet: Scene as Occupancy
https://arxiv.org/abs/2306.02851
Apache License 2.0
566 stars 51 forks source link
3d-object-detection 3d-occupancy autonomous-driving

readme

# Occupancy and Flow Challenge **The tutorial of `Occupancy and Flow` track for [CVPR 2024 Autonomous Grand Challenge](https://opendrivelab.com/challenge2024).**

Introduction

Understanding the 3D surroundings including the background stuffs and foreground objects is important for autonomous driving. In the traditional 3D object detection task, a foreground object is represented by the 3D bounding box. However, the geometrical shape of the object is complex, which can not be represented by a simple 3D box, and the perception of the background stuffs is absent. The goal of this task is to predict the 3D occupancy of the scene. In this task, we provide a large-scale occupancy benchmark based on the nuScenes dataset. The benchmark is a voxelized representation of the 3D space, and the occupancy state and semantics of the voxel in 3D space are jointly estimated in this task. The complexity of this task lies in the dense prediction of 3D space given the surround-view images.

News

[!TIP] :ice_cube: We release a 3D occupancy synthetic dataset LightwheelOcc, with dense occupancy and depth label and realistic sensor configuration simulating nuScenes dataset. Check it out!

Table of Contents

Task Definition

Given images from multiple cameras, the goal is to predict the semantics and flow of each voxel grid in the scene. The paticipants are required to submit their prediction on nuScenes OpenOcc test set.

Rules for Occupancy and Flow Challenge

(back to top)

Evaluation Metrics

Leaderboard ranking for this challenge is by the Occupancy Score. It consists of two parts: Ray-based mIoU, and absolute velocity error for occupancy flow.

The implementation is here: projects/mmdet3d_plugin/datasets/ray_metrics.py

Ray-based mIoU

We use the well-known mean intersection-over-union (mIoU) metric. However, the elements of the set are now query rays, not voxels.

Specifically, we emulate LiDAR by projecting query rays into the predicted 3D occupancy volume. For each query ray, we compute the distance it travels before it intersects any surface. We then retrieve the corresponding class label and flow prediction.

We apply the same procedure to the ground-truth occupancy to obtain the groud-truth depth, class label and flow.

A query ray is classified as a true positive (TP) if the class labels coincide and the L1 error between the ground-truth depth and the predicted depth is less than either a certain threshold (e.g. 2m).

Let $C$ be he number of classes.

$$ mIoU=\frac{1}{C}\displaystyle \sum_{c=1}^{C}\frac{TP_c}{TP_c+FP_c+FN_c}, $$

where $TP_c$ , $FP_c$ , and $FN_c$ correspond to the number of true positive, false positive, and false negative predictions for class $c_i$.

We finally average over distance thresholds of {1, 2, 4} meters and compute the mean across classes.

For more details about this metric, please refer to the technical report.

AVE for Occupancy Flow

Here we measure velocity errors for a set of true positives (TP). We use a threshold of 2m distance.

The absolute velocity error (AVE) is defined for 8 classes ('car', 'truck', 'trailer', 'bus', 'construction_vehicle', 'bicycle', 'motorcycle', 'pedestrian') in m/s.

Occupancy Score

The final occupancy score is defined to be a weighted sum of mIoU and mAVE. Note that the velocity errors are converted to velocity scores as max(1 - mAVE, 0.0). That is,

OccScore = mIoU * 0.9 + max(1 - mAVE, 0.0) * 0.1

(back to top)

OpenOcc Dataset

Basic Information

| Type | Info | | :----: | :----: | | train | 28,130 | | val | 6,019 | | test | 6,008 | | cameras | 6 | | voxel size | 0.4m | | range | [-40m, -40m, -1m, 40m, 40m, 5.4m] | | volume size | [200, 200, 16] | | #classes | 0 - 16 |

Download

  1. Download the nuScenes dataset and put in into data/nuscenes

  2. Download our openocc_v2.1.zip and infos.zip from OpenDataLab or Google Drive

  3. Unzip them in data/nuscenes

Hierarchy

The hierarchy of folder data/nuscenes is described below:

nuscenes
├── maps
├── nuscenes_infos_train_occ.pkl
├── nuscenes_infos_val_occ.pkl
├── nuscenes_infos_test_occ.pkl
├── openocc_v2
├── samples
├── v1.0-test
└── v1.0-trainval

Known Issues

(back to top)

Baseline

We provide a baseline model based on BEVFormer.

Please refer to getting_started for details.

(back to top)

Submission

Submission format

The submission must be a single dict with the following structure:

submission = {
    'method': '',                           <str> -- name of the method
    'team': '',                             <str> -- name of the team, identical to the Google Form
    'authors': ['']                         <list> -- list of str, authors
    'e-mail': '',                           <str> -- e-mail address
    'institution / company': '',            <str> -- institution or company
    'country / region': '',                 <str> -- country or region, checked by iso3166*
    'results': {
        [token]: {                          <str> -- frame (sample) token
            'pcd_cls'                       <np.ndarray> [N] -- predicted class ID, np.uint8,
            'pcd_dist'                      <np.ndarray> [N] -- predicted depth, np.float16,
            'pcd_flow'                      <np.ndarray> [N, 2] -- predicted flow, np.float16,
        },
        ...
    }
}

Below is an example of how to save the submission:

import pickle, gzip

with gzip.open('submission.gz', 'wb', compresslevel=9) as f:
    pickle.dump(submission, f, protocol=pickle.HIGHEST_PROTOCOL)

We provide example scripts based on mmdetection3d to generate the submission file, please refer to baseline for details.

(back to top)

Working with your own codebase

We understand that many participants may use your own codebases. Here, we provide a simple standlone package that converts your occupancy predictions to the submission format. Please follows the steps below:

  1. Save the prediction results on nuScenes OpenOcc val locally, in the same format as the occupancy ground truth.
  2. Perform ray projection locally and save the projection results. 
    cd tools/ray_iou
python ray_casting.py --pred-root your_prediction
  3. Test whether the evaluation on nuScenes OpenOcc val meets expectations locally. 
    python metric.py --pred output/my_pred_pcd.gz --gt output/nuscenes_infos_val_occ_pcd.gz
  4. Save and project the prediction results of nuScenes OpenOcc test according to steps 1 and 2, and upload them to the competition server.

License and Citation

If you use the challenge dataset in your paper, please consider citing OccNet with the following BibTex:

@article{sima2023_occnet,
    title={Scene as Occupancy},
    author={Chonghao Sima and Wenwen Tong and Tai Wang and Li Chen and Silei Wu and Hanming Deng and Yi Gu and Lewei Lu and Ping Luo and Dahua Lin and Hongyang Li},
    year={2023},
    eprint={2306.02851},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

If you use RayIoU as the evaluation metric, please consider citing the following BibTex:

@misc{liu2024fully,
    title={Fully Sparse 3D Occupancy Prediction}, 
    author={Haisong Liu and Yang Chen and Haiguang Wang and Zetong Yang and Tianyu Li and Jia Zeng and Li Chen and Hongyang Li and Limin Wang},
    year={2024},
    eprint={2312.17118},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

This dataset is under CC BY-NC-SA 4.0 license. Before using the dataset, you should register on the website and agree to the terms of use of the nuScenes. All code within this repository is under Apache 2.0 License.

(back to top)