Roadmap of MMDetection

[hellock]

We keep this issue open to collect feature requests from users and hear your voice. Our monthly release plan will be updated in different issues.

In this issue, you can either:

1. Suggest a new feature by leaving a comment.
2. Vote for a feature request with :+1: or be against with :-1:. (Remember that developers are busy and cannot respond to all feature requests, so vote for your most favorable one!)
3. Tell us that you would like to help implement one of the features in the list or review the PRs. (This is the greatest things to hear about!)

We also released our TODO list on the project page. Most of the TODO items are described in their corresponding issues (those labeled by Dev-RD) with detailed requirement documentation. Feel free to leave a message in the issue of any item and create a PR if you are interested in any of the item.

[daavoo]

I would be interested in adding support for EfficientNet (backbone, #764) and EfficientDet (detector, #1827) architectures along with trained models.

I will also be willing to contribute this feature myself.

[mmeendez8]

It would be interesting to add batch inference support. I've seen a couple of issues related with this (#2703 #1833 #1659) and some details in source code and it looks like it has been left as future work (see https://github.com/open-mmlab/mmdetection/blob/master/mmdet/models/detectors/base.py#L118) so it could be a good moment.

I'd be glad to help with this too.

[hiyyg]

How about adding the support for yolov4?

[leehao178]

Hi authors Can you add DOTA dataset support? ex:more bbox parameters! https://captain-whu.github.io/DOTA/dataset.html

Thank you so much!!

[zehuichen123]

Any plans about knowledge distillation for object detection, for example, fitnet (https://github.com/TuSimple/simpledet/blob/master/models/KD/README.md), so that users can customize their distillation loss functions based on their requirements.

[hellock]

I would be interested in adding support for EfficientNet (backbone, #764) and EfficientDet (detector, #1827) architectures along with trained models.

I will also be willing to contribute this feature myself.

Looking forward to it. Is there expected time for EfficientNet and EfficientDet? We need to have a plan whether to release it in V2.2 (June 30) or V2.3 (July 31).

[hellock]

It would be interesting to add batch inference support. I've seen a couple of issues related with this (#2703 #1833 #1659) and some details in source code and it looks like it has been left as future work (see https://github.com/open-mmlab/mmdetection/blob/master/mmdet/models/detectors/base.py#L118) so it could be a good moment.

I'd be glad to help with this too.

Thanks! We have already created a PR (#1833) to support batch inference a few months ago, and will continue working on it to make it compatible with V2.0. You are still welcome to contribute any other new features.

[hellock]

How about adding the support for yolov4?

If there is no community contributors helping with that, we will firstly add YOLO v3 in V2.3 (July 31).

[ZwwWayne]

Hi authors Can you add DOTA dataset support? ex:more bbox parameters! https://captain-whu.github.io/DOTA/dataset.html

Think you so much!!

For now we do not have plan for that, but there have been mmdet-based project doing that. You might be interested in https://github.com/dingjiansw101/AerialDetection.

[daavoo]

I would be interested in adding support for EfficientNet (backbone, #764) and EfficientDet (detector, #1827) architectures along with trained models. I will also be willing to contribute this feature myself.

Looking forward to it. Is there expected time for EfficientNet and EfficientDet? We need to have a plan whether to release it in V2.2 (June 30) or V2.3 (July 31).

I think that probably EfficientNet for version v2.2 and EfficientDet for v2.3

[JosonChan1998]

Hi ! authors Any plan about AC-FPN ? (https://arxiv.org/pdf/2005.11475.pdf

[SuzaKrish]

Could you add some help about how to actually build modules out of these existing ones?

[ZwwWayne]

Hi @SuzaKrish , Tutorials have been already added here. Is there anything unclear or not sufficient?

[SuzaKrish]

Thanks! So in the tutorials, you have mentioned how the base is supposed to be formed right? how do we then call it finally to run the model? Also, while going over the config files, I had a doubt as to where to find these modules in the main repo page. Could you probably include a part in the readme describing which folders contain which components(like head, model structure, base etc.)

[ZwwWayne]

Hi @SuzaKrish , This documentation might help you.

[hiyyg]

How about supporting coco-like evaluation for custom datasets without coco json annotation file? For instance, by converting the custom dataset to coco format internally during evaluation like https://github.com/facebookresearch/detectron2/blob/master/detectron2/evaluation/coco_evaluation.py#L72.

[SuzaKrish]

documentation

Thank you! This is helpful! :) @ZwwWayne

[shawn-xj]

Is there any plan for supporting some light-weight networks like BiSeNet, DFANet?

[JIEMIN1995]

Any plans to support light-weight backbone such as mobilenetV2, mobilenetV3 ?

[ElectronicElephant]

How about adding the support for yolov4?

If there is no community contributors helping with that, we will firstly add YOLO v3 in V2.3 (July 31).

Hello, @WenqiangX and I are glad to help implement the YOLO v3, and perhaps v4 as well, if we feel good. We are from MVIG, SJTU and have spent quite a little time studying all kinds of YOLOv3 implementation, especially the one from gluon-cv, which is probably one of the best re-implementation of YOLOv3.

Basically, we plan to continue the work of #1695 . From a big picture, we plan to do the following:

1. Refactor the backbone with your new ConvModule (as mentioned in https://github.com/open-mmlab/mmdetection/pull/1695#issuecomment-598038394)
2. Solve the problem of not being able to use Distribute Training
3. Introduce some great ideas from gluon-cv

If you are glad with us, can you manage your time to check if there is any major defect in #1695, so that we can save some review time in the future?

BTW, I don't know if the license from Western Digital company will be a big issue.

[hiyyg]

Any plan on adding vovnet backbone, such as https://github.com/aim-uofa/AdelaiDet/tree/master/configs/FCOS-Detection/vovnet?

[hellock]

How about adding the support for yolov4?

If there is no community contributors helping with that, we will firstly add YOLO v3 in V2.3 (July 31).

Hello, @WenqiangX and I are glad to help implement the YOLO v3, and perhaps v4 as well, if we feel good. We are from MVIG, SJTU and have spent quite a little time studying all kinds of YOLOv3 implementation, especially the one from gluon-cv, which is probably one of the best re-implementation of YOLOv3.

Basically, we plan to continue the work of #1695 . From a big picture, we plan to do the following:

1. Refactor the backbone with your new ConvModule (as mentioned in #1695 (comment))
2. Solve the problem of not being able to use Distribute Training
3. Introduce some great ideas from gluon-cv

If you are glad with us, can you manage your time to check if there is any major defect in #1695, so that we can save some review time in the future?

BTW, I don't know if the license from Western Digital company will be a big issue.

Thanks and glad to know you are willing to help! We can have further discussion in that PR and may expect YOLOv3 in V2.3. The copyright is ok as if it is licensed under Apache-2.0.

[chuong98]

Can we add SpineNet (CVPR2020)? The code is provided by @lucifer443 https://github.com/lucifer443/SpineNet-Pytorch As said in the repo, I don't have enough servers to train it, but willing to adjust the code and pull request.

I only trained with protocol B. Training SpineNet takes lots of time, for example I took 7 days to train SpineNet-49 with 8 TITAN V gpus.

[ZwwWayne]

Can we add SpineNet (CVPR2020)? The code is provided by @lucifer443 https://github.com/lucifer443/SpineNet-Pytorch As said in the repo, I don't have enough servers to train it, but willing to adjust the code and pull request.

I only trained with protocol B. Training SpineNet takes lots of time, for example I took 7 days to train SpineNet-49 with 8 TITAN V gpus.

PRs are welcomed.

[tianq01]

any plan for ONNX support? e.g. two stage faster-rcnn thanks.

[mathmanu]

"Bag of Freebies for Training Object Detection Neural Networks"

Describe the feature Is it possible to support the training improvements described in the following paper: "Bag of Freebies for Training Object Detection Neural Networks" Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li https://arxiv.org/pdf/1902.04103.pdf

Motivation It seems the "Bag of Freebies" provide significant accuracy improvement (+5%) in a single shot detector such as YOLOv3. It improves Faster-RCNN by upto 1.7%. It is likely that these features will improve other detectors as well.

Related resources https://medium.com/apache-mxnet/gluoncv-0-3-a-new-horizon-564326364e16 https://gluon-cv.mxnet.io/model_zoo/detection.html

Additional context I think implementing these features would give accuracy lift to several object detectors in mmdetection.

Note: Added here as per the suggetion in https://github.com/open-mmlab/mmdetection/issues/3124

Thanks,

[mathmanu]

"Objects as Points (CenterNet)" is a popular, high accuracy Object Detector.

"Objects as Points" Xingyi Zhou, Dequan Wang, Philipp Krähenbühl https://arxiv.org/abs/1904.07850 Source code: https://github.com/xingyizhou/CenterNet (This detector is referred to as CenterNet in the paper. But there is another detector that calls itself CenterNet - so I am using both names to avoid confusion)

In CVPR2020, the second place solution in the 2D Object detection track of the Waymo open challenges used this detector as one component: https://waymo.com/open/challenges/ https://waymo.com/open/challenges/2d-detection/ "2nd Place Solution for Waymo Open Dataset Challenge - 2D Object Detection" Sijia Chen∗ Yu Wang∗ Li Huang Runzhou Ge Yihan Hu Zhuangzhuang Ding Jie Liao, Horizon Robotics Inc. https://arxiv.org/pdf/2006.15507.pdf

It is also worth noting that this CenterNet inspired AFDet which in-turn was the basis for in the 1st place Solution for 3D Object Detection: "1st Place Solution for Waymo Open Dataset Challenge - 3D Detection and Domain Adaptation" Zhuangzhuang Ding∗ Yihan Hu∗ Runzhou Ge∗ Li Huang Sijia Chen Yu Wang Jie Liao Horizon Robotics https://arxiv.org/pdf/2006.15505.pdf "AFDet: Anchor Free One Stage 3D Object Detection" Runzhou Ge∗ Zhuangzhuang Ding∗ Yihan Hu∗ Yu Wang Sijia Chen Li Huang Yuan Li Horizon Robotics https://arxiv.org/pdf/2006.12671.pdf

So, having this "Objects as Points (CenterNet)" Detector in mmdetection is highly desirable. Kindly add it if possible.

[YAOYI626]

"Objects as Points (CenterNet)" is a popular, high accuracy Object Detector.

Objects as Points Xingyi Zhou, Dequan Wang, Philipp Krähenbühl https://arxiv.org/abs/1904.07850 Source code: https://github.com/xingyizhou/CenterNet (This detector is referred to as CenterNet in the paper. But there is another detector that calls itself CenterNet - so I am using both names to avoid confusion)

In CVPR2020, the second place solution in the 2D Object detection track of the Waymo open challenges used this detector as one component: https://waymo.com/open/challenges/ https://waymo.com/open/challenges/2d-detection/

2nd Place Solution for Waymo Open Dataset Challenge - 2D Object Detection Sijia Chen∗ Yu Wang∗ Li Huang Runzhou Ge Yihan Hu Zhuangzhuang Ding Jie Liao, Horizon Robotics Inc. https://arxiv.org/pdf/2006.15507.pdf

Having this "Objects as Points (CenterNet)" Detector in mmdetection is highly desirabale. Kindly add it if possible.

hey @mathmanu maybe this PR will help you. But it is not offically supported and only work with mmdet v1.x. Would you mind merging it with newer verison ? @hellock @ZwwWayne I strongly believe our community want this one detector in the model zoo.

[ZwwWayne]

Hi @mathmanu , @mathmanu , Thanks for your kind suggestion. Due to limited developers and resources, we are not going to implement those methods in the near future, PRs are welcome.

[ElectronicElephant]

Hi @hellock @xvjiarui ,

The main job of our YOLOv3 implementation was done several days ago. It would be nice if you can manage your time to review the code. Also, I don't have many vid cards, so I haven't tested it with other backbones like ResNet yet. (But it should work in my design.)

Btw, I think if mm-detection is aimed at both academy and production, then it should contain both light-weight and heavy-weight backbones / models. Would you like me to add YOLOv3-tiny, after YOLOv3 is merged?

[hellock]

Hi @hellock @xvjiarui ,

The main job of our YOLOv3 implementation was done several days ago. It would be nice if you can manage your time to review the code. Also, I don't have many vid cards, so I haven't tested it with other backbones like ResNet yet. (But it should work in my design.)

Btw, I think if mm-detection is aimed at both academy and production, then it should contain both light-weight and heavy-weight backbones / models. Would you like me to add YOLOv3-tiny, after YOLOv3 is merged?

Thanks for your great work! The review is ongoing. YOLOv3-tiny is definitely favorable.

[zhjw0927]

Please release a TT100K benchmark. For anchor and augmentation, not have a benchmark for following research.

[hyz-xmaster]

I would suggest improving the training performance stability. I have ran into the same problem with #2773 when training the detectors on the coco dataset. There is generally a maximum of +0.2 or -0.2 performance gap even using the same config file and same seed. This is a bit annoying because you do not know whether the performance gain or drop is due to better parameters or just some randomness.

[mathmanu]

I thought of bringing the following announcement to your notice, as we are discussing features to be included:

TensorFlow 2 meets the Object Detection API https://blog.tensorflow.org/2020/07/tensorflow-2-meets-object-detection-api.html "Over the last year we’ve been migrating our TF Object Detection API models to be TensorFlow 2.... A suite of TF2 compatible (Keras-based) models; ...., as well as a few new architectures for which we will only maintain TF2 implementations: (1) CenterNet - a simple and effective anchor-free architecture based on the recent Objects as Points paper by Zhou et al, and (2) EfficientDet ...."

I think it shows how important and how much awaited these detectors are: Objects as Points and EfficientDet.

[manhongnie]

I hope you will increase your support for onnx, otherwise you will not be able to deploy with mmdetection. If you have succeeded in the steps of onnx, please provide the onnx model

[hellock]

I thought of bringing the following announcement to your notice, as we are discussing features to be included:

TensorFlow 2 meets the Object Detection API https://blog.tensorflow.org/2020/07/tensorflow-2-meets-object-detection-api.html "Over the last year we’ve been migrating our TF Object Detection API models to be TensorFlow 2.... A suite of TF2 compatible (Keras-based) models; ...., as well as a few new architectures for which we will only maintain TF2 implementations: (1) CenterNet - a simple and effective anchor-free architecture based on the recent Objects as Points paper by Zhou et al, and (2) EfficientDet ...."

I think it shows how important and how much awaited these detectors are: Objects as Points and EfficientDet.

We have heard the voice from the community for CenterNet, and will increase the priority in our roadmap. Hopefully we will introduce it to mmdet V2.4.

[michaelschleiss]

I would love to see Test Time Augmentation for Single Stage Detectors. #509

[zhongqiu1245]

Hi, dear authors Thank you for your amazing job!

Could you support BorderDet? The original single-point feature can be directly optimized by using the border features, and the SOTA target detection algorithm BorderDet is proposed based on BorderAlign

Motivation Ex1. Many sliding-window object detectors(such as FCOS,SSD,RetinaNet) adopte the feature maps on the point of the grid to generate the bounding box predictions, but lack the explicit border information for accurate localization. Ex2.If those detectors use the border information, the performance of the detector will be improved. Ex3. There is a recent paper《BorderDet: Border Feature for Dense Object Detection》.This paper gives a detailed explanation of the method of using border information to improve the performance of FCOS(mAP: 38.6 v.s. 41.4) and has achieved good results.

Related resources official code：https://github.com/Megvii-BaseDetection/BorderDet

[zhongqiu1245]

Hi, dear authors Thank you for your amazing job! Could you support FPT?

they use Transformer in FPN and got FPT(FPN + Transformer). FPT improves 8.5% box-AP for object detection and 6.0% mask-AP for instance segmentation over baseline on the MS-COCO test-dev

Related resources official code：https://github.com/ZHANGDONG-NJUST/FPT paper：https://arxiv.org/abs/2007.09451

[bluesky314]

This library is so good. What is stopping wide-spread usage is lack of tutorials for beginners. If you guys spent some time preparing tutorials it would really help the library.

[ZwwWayne]

This library is so good. What is stopping wide-spread usage is lack of tutorials for beginners. If you guys spent some time preparing tutorials it would really help the library.

Hi @bluesky314 , Thanks for your kind advice. For the tutorials and docs, we have made some progress, e.g., documentation with 4 tutorials and colab tutorials. We are still working on that to make it easier for users to start. Therefore, could you be more specific? For example, what tutorials do you think might be valuable but we are missing for now? We will try to complete them in our nearest release plan.

[bluesky314]

Off the top of my head I can think of: Creating Hooks Using Hooks(like tensorboard,wandb, etc) Adding losses Weighting/tweaking losses differently Modifying the train loop

[zeakey]

Any plan for contour-based instance segmentation methods like PolarMask (https://arxiv.org/abs/1909.13226), DeepSnake (https://arxiv.org/abs/2001.01629) and Dense RepPoints (https://arxiv.org/pdf/1912.11473v3)?

[wenmengzhou]

Any plan for jit support? some related issues #1504 #2856

[YAOYI626]

Hey @hellock , just noticed the updated new plans for v2.5. It's super exciting. But there still is no plan for the CenterNet: objects as points? A real pity.

[StetchPlane]

Thanks for the excellent codebase. Do you have any plans to support BorderDet?

The original single-point feature can be directly optimized by using the border features, and the SOTA target detection algorithm BorderDet is proposed based on BorderAlign.

official code：https://github.com/Megvii-BaseDetection/BorderDet

[sanghun3819]

Hi, dear authors Thank you for your amazing job!

Could you support BorderDet? The original single-point feature can be directly optimized by using the border features, and the SOTA target detection algorithm BorderDet is proposed based on BorderAlign

Motivation Ex1. Many sliding-window object detectors(such as FCOS,SSD,RetinaNet) adopte the feature maps on the point of the grid to generate the bounding box predictions, but lack the explicit border information for accurate localization. Ex2.If those detectors use the border information, the performance of the detector will be improved. Ex3. There is a recent paper《BorderDet: Border Feature for Dense Object Detection》.This paper gives a detailed explanation of the method of using border information to improve the performance of FCOS(mAP: 38.6 v.s. 41.4) and has achieved good results.

Related resources official code：https://github.com/Megvii-BaseDetection/BorderDet

[edwardyangxin]

After yolov4, will scaled yolov4 on the road? https://arxiv.org/abs/2011.08036

[zhongqiu1245]

Hi, dear authors Thank you for your amazing job!

Could you support Sparse R-CNN?

Sparse R-CNN:End-to-End Object Detection with Learnable Proposals official code:https://github.com/PeizeSun/SparseR-CNN paper:https://arxiv.org/abs/2011.12450

No anchor, No NMS, No RPN, as better as DETR but faster than DETR, especially APs

[chriss2401]

I would also like to see more onnx support for models, it will make much easier to use trained models in production. Congrats on the good work so far!