Roadmap of MMDetection3d

[hellock]

We keep this issue open to collect feature requests from users and hear your voice. Our monthly release plan is also available here.

You can either:

1. Suggest a new feature by leaving a comment.
2. Vote for a feature request with 👍 or be against with 👎. (Remember that developers are busy and cannot respond to all feature requests, so vote for your most favorable one!)
3. Tell us that you would like to help implement one of the features in the list or review the PRs. (This is the greatest things to hear about!)

V0.7.0 (October-November)

• [x] Switch visualization backend to Open3D
• [ ] More performance on Waymo, nuScenes, SUN RGB-D, and ScanNet dataset

V0.6.0 (September)

• [x] Waymo Dataset (#118)
• [x] Batch Inference (#95)
• [x] CenterPoint (#49, #92)
• [x] 3DSSD (#83, #100, #104)
• [x] H3DNet (#53, #58, #105)

[turboxin]

Hi all, thanks for releasing this wonderful project！

I 'm wondering what's your recent development plan for new models or methods?

[thuyngch]

In the previous version (PCDet), I see that PV-RCNN is supported. However, it is not supported now. So, I hope to see PV-RCNN again.

[hellock]

In the previous version (PCDet), I see that PV-RCNN is supported. However, it is not supported now. So, I hope to see PV-RCNN again.

PCDet and mmdet3d are two different codebases. You may refer to PCDet for PV-RCNN now, and we will support it in mmdet3d soon.

[SilvioGiancola]

Thanks for the development of MMDetection3d. I would appreciate the support for the Waymo Dataset.

[chuong98]

Thanks for the opensource. Can we support 3DSSD https://github.com/Jia-Research-Lab/3DSSD? It seems achieve both good speed and high AP. Thank you.

[zehanzhang]

Can we support SA-SSD https://github.com/skyhehe123/SA-SSD？The AP and speed is also good.

[hhcNWPU]

Hello, this code can visualize the 3D detection box in the point cloud, may I ask if the 3D detection box can be visualized in the 2D image，and how do we do that?

[linchunmian]

Hi. all. I hope to see the implementation of pseudo LiDAR in mmdetection3d. And also, I think more visualization of data processing and detection result are required, too. Thanks.

[AllenPeng0209]

Hi, all. the current strongest backbone SPVNAS could be a good thing to implement. Also, it's new library torchsparse is much faster than the current library.

https://github.com/mit-han-lab/torchsparse https://github.com/mit-han-lab/e3d/blob/master/spvnas/core/models/semantic_kitti/spvnas.py

[PersistentXQ]

Hi, all. If I wanna add a feature to dataset (for example, besides (x,y,z,intensity), I wanna add another feature to it and make the data become (x,y,z,intensity,another featrue).), could you guys provides some interfaces or functions or sth?

[ZwwWayne]

Hi, all. If I wanna add a feature to dataset (for example, besides (x,y,z,intensity), I wanna add another feature to it and make the data become (x,y,z,intensity,another featrue).), could you guys provides some interfaces or functions or sth?

We recommend to do this in the model part. You can implement a new module to process the point data following the guide here

[PersistentXQ]

Hi, all. If I wanna add a feature to dataset (for example, besides (x,y,z,intensity), I wanna add another feature to it and make the data become (x,y,z,intensity,another featrue).), could you guys provides some interfaces or functions or sth?

We recommend to do this in the model part. You can implement a new module to process the point data following the guide here

ZwwWayne, thank you very much for your answer. I think I asked a wrong question… The new kitti-like dataset (x,y,z,intensity,another feature) have already been made. The question is how I can train the model with my dataset. I tried to change the "load_dims" and "use_dims" in configs from 4 to 5, but some errors were threw… After I fixed these errors, another errors were threw again… Now the newest error is Runtime Error: size mismatch.

[ZwwWayne]

Hi, all. If I wanna add a feature to dataset (for example, besides (x,y,z,intensity), I wanna add another feature to it and make the data become (x,y,z,intensity,another featrue).), could you guys provides some interfaces or functions or sth?

We recommend to do this in the model part. You can implement a new module to process the point data following the guide here

ZwwWayne, thank you very much for your answer. I think I asked a wrong question… The new kitti-like dataset (x,y,z,intensity,another feature) have already been made. The question is how I can train the model with my dataset. I tried to change the "load_dims" and "use_dims" in configs from 4 to 5, but some errors were threw… After I fixed these errors, another errors were threw again… Now the newest error is Runtime Error: size mismatch.

Feel free to create a new issue to describe your issues following the reimplementation issues template if you need some help. This issue is mainly about the roadmap.

[ZwwWayne]

Hi, all. the current strongest backbone SPVNAS could be a good thing to implement. Also, it's new library torchsparse is must faster than the current library.

https://github.com/mit-han-lab/torchsparse https://github.com/mit-han-lab/e3d/blob/master/spvnas/core/models/semantic_kitti/spvnas.py

Thanks for the suggestion. We will investigate these repos and may change the dependency if necessary.

[YoushaaMurhij]

Can you add support for sematic-kitti ?

[chasingw]

any plan for ONNX support?

[ZwwWayne]

any plan for ONNX support?

This is a planned feature, but we may not support it very soon. So PRs are also welcomed.

[qinsimon]

Thanks for your job. And it would be nice if cuda11.1/torch1.7 are compatible

[chakri1804]

Can you add support for Pandaset ?

[filaPro]

Do you have plans to release the code for your FCOS3D on NuScenes in near future?

[Tai-Wang]

Do you have plans to release the code for your FCOS3D on NuScenes in near future?

Yes. More information could be referred to #247 .

[Jsijon]

Can you support Cirrus dataset ? https://arxiv.org/abs/2012.02938

[Wuziyi616]

Hello, this code can visualize the 3D detection box in the point cloud, may I ask if the 3D detection box can be visualized in the 2D image，and how do we do that?

Hi, this multi-modality visualization has been supported recently. Check out our demo and how to use for it!

[xinghuokang]

any plan for ONNX support?

This is a planned feature, but we may not support it very soon. So PRs are also welcomed.

when to support onnx?

[zehuichen123]

Any plans for Range-View 3D detection models? such as RCD, RangeDet, RSN(CVPR2021), ToThePoint(CVPR2021).

[ZwwWayne]

Any plans for Range-View 3D detection models? such as RCD, RangeDet, RSN(CVPR2021), ToThePoint(CVPR2021).

We are actually planning for it. Do you have any recommendations?

[zehuichen123]

Any plans for Range-View 3D detection models? such as RCD, RangeDet, RSN(CVPR2021), ToThePoint(CVPR2021).

We are actually planning for it. Do you have any recommendations?

I think RSN is a great choice though it does not ONLY use range view compared to RangeDet, ToThePoint or RCD :)

[afonso-sousa]

I would love to see more Mono3D, Stereo3D, and Camera+LiDAR detectors. Maybe RTM3D, YoloStereo3D and Frustum-PointNet.

[fortkord]

I would really appreciate seeing the implementation of MoCa since it has been announced quite some time ago, as well as more (real-time) camera-lidar pipelines. Happy to list some examples if necessary

[victoryc]

It will be nice to support 3D semantic labeling and 3D instance segmentation, along with sparse convolution functionality. Some examples of items that would be good to have are:

• 3D Unet
• PointGroup
• OccuSeg
• PointTransformer

[fcakyon]

It would be nice to support FCAF3D (SOTA in ScanNet, SUN RGB-D, S3DIS)

Source code is available on top of mmdet3d base: https://github.com/samsunglabs/fcaf3d

[PersistentXQ]

这是来自QQ邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。徐谦

[little-bookworm]

Do you have plans for model deployment and conversion？

[poddar414]

pointPainting

[PersistentXQ]

这是来自QQ邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。徐谦

[nacayu]

It would be nice if mmdet3d could incorporate radar information like lidar, Our lab, and many radar researcher also need a good platform to employ radar and camera fusion and so on, thanks.

[PersistentXQ]

这是来自QQ邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。徐谦

[xiaoyangyang123]

你好，请问dgcnn的内容是删除了嘛？什么时候重新上线呢？

[PersistentXQ]

这是来自QQ邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。徐谦

[sxfzzz]

Would you consider RPVNet? RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation

[PersistentXQ]

这是来自Foxmail邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。徐谦

[xiaoyangyang123]

你会考虑RPVNet吗？ RPVNet：用于激光雷达点云分段的深度高效范围-点-体素融合网络

有考虑用point transformer进行尝试，您说的这个我还没有细看。主要是想将新内容用在dgcnn上进行尝试提升，不知您有什么好的建议？

[CarlHuangNuc]

deep fusion and bevfusion

[PersistentXQ]

这是来自Foxmail邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。

[YoushaaMurhij]

Support radar data in nuScenes!

[PersistentXQ]

这是来自Foxmail邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。

[zhanghang1989]

Suggested by @JingweiZhang12 . Add request for Cube RCNN implementation. It would be great if mmdetection3d could support Cube RCNN from Omni3d paper, so that we can easily compare across different method.

[PersistentXQ]

这是来自Foxmail邮箱的自动回复邮件。   您好，您的来信已收到，我将尽快给您回复。

[schneidersebastian]

It would be nice if mmdet3d could incorporate radar information like lidar, Our lab, and many radar researcher also need a good platform to employ radar and camera fusion and so on, thanks.

Any updates on that?

[aFickle]

为什么我就是实现不了目标检测的demo，谁能帮帮我