search

nutonomy / nuscenes-devkit

The devkit of the nuScenes dataset.
https://www.nuScenes.org
Other
2.25k stars 623 forks source link

Performance Reference on nuImages by MMDetection3D #474

Closed ZwwWayne closed 3 years ago

ZwwWayne commented 3 years ago

Hi developers,

Thanks for your great work on both nuScenes & nuImages datasets. To catch up with the release of nuImages, MMDetection3D just supported nuImages dataset and released Mask R-CNN/Cascade Mask R-CNN pre-trained models and their corresponding results here.

So firstly I wish you and the communities who are interested in nuScenes/nuImages know this news and try out the pre-trained models. They could not only serve as good pre-trained models but also references of the models' performance.

The second thing is a question about the implementation, especially about the class definition. The current implementation in MMDetection3D uses the same class mapping as it is in nuScenes. Thus, it maps the original about 30 categories to 10 classes and perform instance segmentation/object detection. I am not sure whether this is a suitable practice so I am seeking your opinions. So maybe this issue is also created for discussion about the implementation on nuImages dataset.

BTW, direct PRs, suggestions, and collaborations for MMDet3D are also welcomed.

Thanks.

holger-motional commented 3 years ago

Hi @ZwwWayne. That's wonderful news! To spread the word, can we create a news item on https://www.nuscenes.org/ and https://www.nuscenes.org/publications and use the MMDetection3D logo?

Regarding the evaluation protocol: I agree that we should be using the same 10 classes as in the 3d object detection challenge. We'll setup a page to provide some more details how to evaluate object detection and instance segmentation on nuImages.

The page for nuImages (https://github.com/open-mmlab/mmdetection3d/tree/master/configs/nuimages) is very useful. Perhaps you could add some example images to show the predictions vs. the ground-truth? That might help your users get a feeling for the type of data and the quality of the model.

ZwwWayne commented 3 years ago

Hi @holger-nutonomy ,

As for the news item about MMDet3D, yes you can. And thank you for helping us to spread the word.

Regarding the evaluation protocol: that's great! We are glad that MMDetection3D and the models are beneficial. We will also keep update with your official page and keep our model consistent with your protocols.

As for the model page. Thanks for your kind and valuable suggestion, we will update the page in the following weeks with HTC models (https://github.com/open-mmlab/mmdetection3d/pull/155), and update some example images to show the predictions vs. the ground-truth.

holger-motional commented 3 years ago

Closing this now. We will update our homepage shortly.