[LION: Linear Group RNN for 3D Object Detection in Point Clouds](https://arxiv.org/abs/2407.18232)
[Zhe Liu](https://happinesslz.github.io) 1,* ,
[Jinghua Hou](https://github.com/AlmoonYsl) 1,* ,
[Xinyu Wang](https://github.com/deepinact) 1,* ,
[Xiaoqing Ye](https://shuluoshu.github.io) 3,
[Jingdong Wang](https://jingdongwang2017.github.io) 3,
[Hengshuang Zhao](https://hszhao.github.io) 2,
[Xiang Bai](https://xbai.vlrlab.net) 1,✉
1 Huazhong University of Science and Technology, 2 The University of Hong Kong, 3 Baidu Inc.
\* Equal contribution, ✉ Corresponding author.
[**Project Page**](https://happinesslz.github.io/projects/LION) | [**NeurIPS 2024**](https://arxiv.org/abs/2407.18232)
🔥 Highlights
-
Strong performance. LION achieves state-of-the-art performance on Waymo, nuScenes, Argoverse V2, and ONCE datasets. 💪
-
Strong generalization. LION can support almost all linear RNN operators including Mamba, RWKV, RetNet, xLSTM, and TTT. Anyone is welcome to verify more linear RNN operators. 😀
-
More friendly. LION can train all models on less 24G GPU memory (i.e., RTX 3090, RTX4090, V100 and A100 are enough to train our LION). 😀
News
- 2024.09.26: LION has been accepted by NeurIPS 2024. 🎉
- 2024.07.25: LION paper released. 🔥
- 2024.07.02: Our new works OPEN and SEED have been accepted by ECCV 2024. 🎉
Results
- Waymo Val Set~(100%)
| Model | mAP/mAPH_L1 | mAP/mAPH_L2 | Vec_L1 | Vec_L2 | Ped_L1 | Ped_L2 | Cyc_L1 | Cyc_L2 | Config |
|---|---|---|---|---|---|---|---|---|---|
| LION-RetNet | 80.9/78.8 | 74.6/72.7 | 79.0/78.5 | 70.6/70.2 | 84.6/80.0 | 77.2/72.8 | 79.0/78.0 | 76.1/75.1 | config |
| LION-RWKV | 81.0/79.0 | 74.7/72.8 | 79.7/79.3 | 71.3/71.0 | 84.6/80.0 | 77.1/72.7 | 78.7/77.7 | 75.8/74.8 | config |
| LION-Mamba | 81.4/79.4 | 75.1/73.2 | 79.5/79.1 | 71.1/70.7 | 84.9/80.4 | 77.5/73.2 | 79.7/78.7 | 76.7/75.8 | config |
| LION-Mamba-L | 82.1/80.1 | 75.9/74.0 | 80.3/79.9 | 72.0/71.6 | 85.8/81.4 | 78.5/74.3 | 80.1/79.0 | 77.2/76.2 | config |
Note: You could reduce the training epochs from 24 to 12~(the performance gap is within 1 mAP/mAPH) or reduce the 100% training to 20% training sets.
- nuScenes
| Model | Split | Epoch | CBGS | NDS | mAP | Config | Download (Baidu Pan) | Download (Google Drive) |
|---|---|---|---|---|---|---|---|---|
| LION-RetNet | Val | 36 | False | 71.9 | 67.3 | config | nus_retnet.pth (ksmp) | nus_retnet.pth |
| LION-RWKV | Val | 36 | False | 71.7 | 66.8 | config | ||
| LION-Mamba | Val | 36 | False | 72.1 | 68.0 | config | nus_mamba.pth (2tvc) | nus_mamba.pth |
| LION-Mamba | Val | 48 | False | 72.3 | 68.2 | config | ||
| LION-Mamba | Test | 36 | False | 73.9 | 69.8 |
Note: Our model on nuScenes does not use CBGS for training more time and without any test-time augmentation or model ensembling! For obtaining more stable and better performance, you could try to train more time~(e.g., 48 epochs)
- Argoverse V2 Val Set
| Model | mAP | Config | Download (Baidu Pan) | Download (Google Drive) |
|---|---|---|---|---|
| LION-RetNet | 40.7 | config | argov2_retnet.pth (yghm) | argov2_retnet.pth |
| LION-RWKV | 41.1 | config | argov2_rwkv.pth (cr4e) | argov2_rwkv.pth |
| LION-Mamba | 41.5 | config | argov2_mamba.pth (k63i) | argov2_mamba.pth |
- ONCE Val Set
| Model | Vehicle | Pedestrian | Cyclist | mAP | Config | Download |
|---|---|---|---|---|---|---|
| LION-RetNet | 78.1 | 52.4 | 68.3 | 66.3 | config | |
| LION-RWKV | 78.3 | 50.6 | 68.4 | 65.8 | config | |
| LION-Mamba | 78.2 | 53.2 | 68.5 | 66.6 | config |
Quick Validation
- We provide some examples of LION models on KITTI dataset for quick validation of any Linear RNN operators.
- Here, we provide the results of moderate difficulty for LION with RetNet, RWKV, Mamba, xLSTM, and TTT.
- Anyone is welcome to verify more linear RNN operators. 😀
| Model | Car | Pedestrian | Cyclist | Config | Download |
|---|---|---|---|---|---|
| LION-TTT | 78.0 | 58.6 | 69.6 | config | |
| LION-xLSTM | 77.9 | 59.3 | 67.4 | config | |
| LION-RetNet | 77.9 | 60.2 | 69.6 | config | |
| LION-Mamba | 78.3 | 60.2 | 68.6 | config | |
| LION-RWKV | 78.3 | 62.2 | 71.2 | config |
Installation
Please refer to INSTALL.md for the installation of LION codebase.
Getting Started
We provide all training&evaluation scripts for training our LION, please refer to tools/
-
Train all models of LION on nuScenes
bash run_train_lion_for_nus.sh -
Train all models of LION on Waymo
bash run_train_lion_for_waymo.sh -
Train all models of LION on Argoverse V2
bash run_train_lion_for_argov2.sh -
Train all models of LION on ONCE
bash run_train_lion_for_once.sh -
Train all models of LION on KITTI
bash run_train_lion_for_kitti.sh
For more details about LION, please refer to GETTING_STARTED.md to learn more usage about LION.
TODO
- [x] Release the paper.
- [x] Release the code of LION on the Waymo.
- [x] Release the code of LION on the nuScenes.
- [x] Release the code of LION on the Argoverse V2.
- [x] Release the code of LION on the ONCE.
- [x] Release the code of LION on the KITTI.
- [x] Release some important checkpoints of LION (nuScenes and Argoverse v2).
- [ ] Support more linear RNNs.
Citation
@article{liu2024lion,
title={LION: Linear Group RNN for 3D Object Detection in Point Clouds},
author={Zhe Liu, Jinghua Hou, Xingyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, Xiang Bai},
journal={Advances in Neural Information Processing Systems},
year={2024}
}Acknowledgements
We thank these great works and open-source repositories: OpenPCDet, DSVT, FlatFormer, HEDNet, Mamba, RWKV, Vision-RWKV, RMT, xLSTM, TTT, and flash-linear-attention.