E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation
Tao Zhang, Shiqing Wei, Shunping Ji
CVPR 2022
Any questions or discussions are welcomed!
An e2ec implementation using mmdetection is complete: e2ec-mmdet. The performance on coco-val is as follows: | Detector | Agu | shcedule | AP | config |
---|---|---|---|---|---|
FCOS | w/o | 1× | 32.9 | config | |
FCOS | w | 4× | 35.1 | config |
Please see INSTALL.md.
We re-tested the speed on a single RTX3090.
Dtataset | AP | Image size | FPS |
---|---|---|---|
SBD val | 59.2 | 512×512 | 59.60 |
COCO test-dev | 33.8 | original size | 35.25 |
KINS val | 34.0 | 768×2496 | 12.39 |
Cityscapes val | 34.0 | 1216×2432 | 8.58 |
The accuracy and inference speed of the contours at different stages on SBD val set. We also re-tested the speed on a single RTX3090.
stage | init | coarse | final | final-dml |
---|---|---|---|---|
AP | 51.4 | 55.9 | 58.8 | 59.2 |
FPS | 101.73 | 91.35 | 67.48 | 59.6 |
The accuracy and inference speed of the contours at different stages on coco val set.
stage | init | coarse | final | final-dml |
---|---|---|---|---|
AP | 27.8 | 31.6 | 33.5 | 33.6 |
FPS | 80.97 | 72.81 | 42.55 | 35.25 |
Download the pretrained model here or Baiduyun(password is e2ec
).
Prepared the COCO dataset according to the INSTALL.md.
Test:
# testing segmentation accuracy on coco val set
python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True
# testing detection accuracy on coco val set
python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --eval bbox
# testing the speed
python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --type speed
# testing the contours of specified stage(init/coarse/final/final-dml)
python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
# testing on coco test-dev set, run and submit data/result/results.json
python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --dataset coco_test
Download the pretrained model here or Baiduyun(password is e2ec
).
Prepared the SBD dataset according to the INSTALL.md.
Test:
# testing segmentation accuracy on SBD
python test.py sbd --checkpoint /path/to/model_sbd.pth
# testing detection accuracy on SBD
python test.py sbd --checkpoint /path/to/model_sbd.pth --eval bbox
# testing the speed
python test.py sbd --checkpoint /path/to/model_sbd.pth --type speed
# testing the contours of specified stage(init/coarse/final/final-dml)
python test.py sbd --checkpoint /path/to/model_sbd.pth --stage coarse
Download the pretrained model here or Baiduyun(password is e2ec
).
Prepared the KINS dataset according to the INSTALL.md.
Test:
Maybe you will find some troules, such as object of type <class 'numpy.float64'> cannot be safely interpreted as an integer
. Please modify the /path/to/site-packages/pycocotools/cooceval.py
. Replace np.round((0.95 - .5) / .05)
in lines 506 and 507 with int(np.round((0.95 - .5) / .05))
.
# testing segmentation accuracy on KINS
python test.py kitti --checkpoint /path/to/model_kitti.pth
# testing detection accuracy on KINS
python test.py kitti --checkpoint /path/to/model_kitti.pth --eval bbox
# testing the speed
python test.py kitti --checkpoint /path/to/model_kitti.pth --type speed
# testing the contours of specified stage(init/coarse/final/final-dml)
python test.py kitti --checkpoint /path/to/model_kitti.pth --stage coarse
Download the pretrained model here or Baiduyun(password is e2ec
).
Prepared the KINS dataset according to the INSTALL.md.
Test:
We will soon release the code for e2ec with multi component detection. Currently only supported for testing e2ec performance on cityscapes dataset.
# testing segmentation accuracy on Cityscapes with coco evaluator
python test.py cityscapesCoco --checkpoint /path/to/model_cityscapes.pth
# with cityscapes official evaluator
python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth
# testing the detection accuracy
python test.py cityscapesCoco \
--checkpoint /path/to/model_cityscapes.pth --eval bbox
# testing the speed
python test.py cityscapesCoco \
--checkpoint /path/to/model_cityscapes.pth --type speed
# testing the contours of specified stage(init/coarse/final/final-dml)
python test.py cityscapesCoco \
--checkpoint /path/to/model_cityscapes.pth --stage coarse
# testing on test set, run and submit the result file
python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth \
--dataset cityscapes_test
Install the Boundary IOU API according boundary iou.
Testing segmentation accuracy with coco evaluator.
Using offline evaluation pipeline.
python /path/to/boundary_iou_api/tools/coco_instance_evaluation.py \
--gt-json-file /path/to/annotation_file.json \
--dt-json-file data/result/result.json \
--iou-type boundary
Download the pretrained model.
Visualize:
# inference and visualize the images with coco pretrained model
python visualize.py coco /path/to/images \
--checkpoint /path/to/model_coco.pth --with_nms True
# you can using other pretrained model, such as cityscapes
python visualize.py cityscapesCoco /path/to/images \
--checkpoint /path/to/model_cityscapes.pth
# if you want to save the visualisation, please specify --output_dir
python visualize.py coco /path/to/images \
--checkpoint /path/to/model_coco.pth --with_nms True \
--output_dir /path/to/output_dir
# visualize the results at different stage
python visualize.py coco /path/to/images \
--checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
# you can reset the score threshold, default is 0.3
python visualize.py coco /path/to/images \
--checkpoint /path/to/model_coco.pth --with_nms True --ct_score 0.1
# if you want to filter some of the jaggedness caused by dml
# please using post_process
python visualize.py coco /path/to/images \
--checkpoint /path/to/model_coco.pth --with_nms True \
--with_post_process True
We have released the code for multi GPU training with ddp.
CUDA_VISIBLE_DEVICES=${gpu_ids} python -m torch.distributed.launch \
--nproc_per_node ${gpu_nums} \
train_net_ddp.py \
--config_file ${dataset} \
--bs ${bs_per_gpu} \
--gpus ${gpu_nums}
# the example of training sbd dataset using 2 gpus
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nproc_per_node 2 \
train_net_ddp.py \
--config_file sbd \
--bs 12 \
--gpus 2
python train_net.py sbd --bs $batch_size
# if you do not want to use dinamic matching loss (significantly improves
# contour detail but introduces jaggedness), please set --dml as False
python train_net.py sbd --bs $batch_size --dml False
python train_net.py kitti --bs $batch_size
python train_net.py cityscapesCoco --bs $batch_size
In fact it is possible to achieve the same accuracy without training so many epochs.
# first to train with adam
python train_net.py coco --bs $batch_size
# then finetune with sgd
python train_net.py coco_finetune --bs $batch_size \
--type finetune --checkpoint data/model/139.pth
If the annotations is in coco style:
Add dataset information to dataset/info.py
.
Modify the configs/coco.py
, reset the train.dataset
, model.heads['ct_hm']
and test.dataset
. Maybe you also need to change the train.epochs
, train.optimizer['milestones']
and so on.
Train the network.
python train_net.py coco --bs $batch_size
If the annotations is not in coco style:
Prepare dataset/train/your_dataset.py
and dataset/test/your_dataset.py
by referring to dataset/train/base.py
and dataset/test/base.py
.
Prepare evaluator/your_dataset/snake.py
by referring to evaluator/coco/snake.py
.
Prepare configs/your_dataset.py
and by referring to configs/base.py
.
Train the network.
python train_net.py your_dataset --bs $batch_size
If you find this project helpful for your research, please consider citing using BibTeX below:
@inproceedings{zhang2022e2ec,
title={E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation},
author={Zhang, Tao and Wei, Shiqing and Ji, Shunping},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={4443--4452},
year={2022}
}
Code is largely based on Deep Snake. Thanks for their wonderful works.