search

jshilong / DDQ

Dense Distinct Query for End-to-End Object Detection (CVPR2023)
Apache License 2.0
244 stars 6 forks source link
detr object-detection

readme

Dense Distinct Query for End-to-End Object Detection (CVPR2023)

By Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, Kai Chen

Updates

21/3/2023: DDQ DETR(12 epoch 52.1 mAP) has been released at _ddq_detr_ branch. Please run git checkout ddq_detr to use it.

20/3/2023: DDQ FCN and DDQ R-CNN has been released at main branch

Abstract:

One-to-one label assignment in object detection has successfully obviated the need of non-maximum suppression (NMS) as a postprocessing and makes the pipeline end-to-end. However, it triggers a new dilemma as the widely used sparse queries cannot guarantee a high recall, while dense queries inevitably bring more similar queries and encounters optimization difficulty. As both sparse and dense queries are problematic, then what are the expected queries in end-to-end object detection? This paper shows that the solution should be Dense Distinct Queries (DDQ). Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments. DDQ blends the advantages of traditional and recent end-to-end detectors and significantly improves the performance of various detectors including FCN, R-CNN, and DETRs. Most impressively, ddq_detr achieves 52.1 AP on MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming all existing detectors in the same setting. DDQ also shares the benefit of end-to-end detectors in crowded scenes and achieves 93.8 AP on CrowdHuman. We hope DDQ can inspire researchers to consider the complementarity between traditional methods and end-to-end detectors.

//: # ()

NOTE

  1. The DDQ FCN/RCNN was implemented based on mmdetection2.0, which has been released in this main branch .

  2. DDQ DETR is implemented based on mmdetection3.0. You need to switch to the _ddqdetr branch of this repo.

All our implementation can be find in projects directory.

Install

DDQ FCN&R-CNN

DDQ FCN and DDQ R-CNN has been fully tested under MMDetection V2.22.0 and MMCV V1.4.7 under torch 1.9.0+cu111, other versions may not be compatible. We include the necessary source code of MMCV and MMDetection in this repo, you can build the MMCV by following command.

cd mmcv-1.4.7
MMCV_WITH_OPS=1 python setup.py build_ext --inplace
ln -s mmcv-1.4.7/mmcv ./
cd ..
export PYTHONPATH=`pwd`:$PYTHONPATH

DDQ DETR

DDQ DETR has been fully tested under MMDetection V3.0.0rc6, MMCV v2.0.0rc4 and MMEngine v0.6.0 under torch 1.9.0+cu111, other versions may not be compatible. We include the necessary source code of MMCV, MMDetection and MMEngine in the _ddqdetr branch of this repo, you just need to build the MMCV by following command.

git check ddq_detr
cd mmcv-2.0.0rc4
MMCV_WITH_OPS=1 python setup.py build_ext --inplace
ln -s mmcv-2.0.0rc4/mmcv ./
cd ..
export PYTHONPATH=`pwd`:$PYTHONPATH

Prepare the dataset

DDQ
├── data
│   ├── coco
│   │   ├── annotations
│   │   │      ├──instances_train2017.json
│   │   │      ├──instances_val2017.json
│   │   ├── train2017
│   │   ├── val2017

Train and Test

8 is the number of gpus.

For slurm

Train

GPUS=8 sh tools/slurm_train.sh partition_name  job_name projects/configs/ddq_fcn/ddq_fcn_r50_1x.py  ./exp/ddq-fcn

Test

GPUS=8 sh tools/slurm_test.sh partition_name  job_name  projects/configs/ddq_fcn/ddq_fcn_r50_1x.py   path_to_checkpoint --eval bbox
For Pytorch distributed

Train

sh tools/train.sh projects/configs/ddq_fcn/ddq_fcn_r50_1x.py 8 ./exp/ddq_fcn

Test

sh tools/test.sh  projects/configs/ddq_fcn/ddq_fcn_r50_1x.py  path_to_checkpoint 8 --eval bbox

Results & Checkpoints

We find that the performance is unstable and may fluctuate by about 0.2 mAP.

Model Backbone Lr schd Augmentation box AP(val) Model log
DDQ FCN R-50 12e Normal 41.5 ckpt log
DDQ FCN R-50 36e DETR 44.8 ckpt log
DDQ R-CNN R-50 12e Normal 44.6 ckpt log
DDQ R-CNN R-50 36e DETR 48.1 ckpt log
Please git checkout ddq_detr and follow the install instruction to use following checkpoint
Model Backbone Lr schd Augmentation box AP(val) Model log
DDQ DETR-4scale R-50 12e DETR 51.3 ckpt log
DDQ DETR-5scale R-50 12e DETR 52.1 ckpt log
DDQ DETR-4scale Swin-L 30e DETR 58.7 ckpt log

Citations

Please cite our paper in your publications if it helps your research:

@InProceedings{Zhang_2023_CVPR,
    author    = {Zhang, Shilong and Wang, Xinjiang and Wang, Jiaqi and Pang, Jiangmiao and Lyu, Chengqi and Zhang, Wenwei and Luo, Ping and Chen, Kai},
    title     = {Dense Distinct Query for End-to-End Object Detection},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {7329-7338}
}