search

nutonomy / nuscenes-devkit

The devkit of the nuScenes dataset.
https://www.nuScenes.org
Other
2.25k stars 623 forks source link

nuImages and nuScenes #692

Closed Di-Gu closed 2 years ago

Di-Gu commented 2 years ago

Hi, Dear Are the nuImages sample data tokens related to the data in nuScenes? Like can I use nuImages data find corresponding annotations in nuScenes?

Regards,

whyekit-motional commented 2 years ago

Hi @Di-Gu, the data in nuImages is independent of the data in nuScenes

jingyibo123 commented 2 years ago

It's really uinfortunate that cam/sensor annotation are splitted in both datsets, Could've facilitated a tons of depth/semantic/multi-modal oriented researches.

whyekit-motional commented 2 years ago

@jingyibo123 nuScenes itself has all sensors for each given sample, so you could potentially do multi-modal research for 3D object detection on it

If you are interested in multi-modal research for 3D segmentation, do check out Panoptic nuScenes (it contains per-point annotations for semantic segmentation, instance segmentation, and tracking)! :smile:

jingyibo123 commented 2 years ago

@jingyibo123 nuScenes itself has all sensors for each given sample, so you could potentially do multi-modal research for 3D object detection on it

If you are interested in multi-modal research for 3D segmentation, do check out Panoptic nuScenes (it contains per-point annotations for semantic segmentation, instance segmentation, and tracking)! smile

Kudos for the lightening-fast reply... Image semantic/static-moving segmentation dataset is labor-intensive and thus rare, for me that's the gold mine in nuImages.. Thanks anyway!

whyekit-motional commented 2 years ago

Just thinking out loud here - a possible way to get 2D semantic segmentation labels on nuScenes could be to:

  1. Train a 2D semantic segmentation model on nuImages / get a 2D semantic segmentation model which has been pretrained on nuImages (e.g. from OpenMMLab)
  2. Use the model to label the image data in nuScenes (i.e. pseudo-labelling of sorts)
  3. Train your 2D instance semantic segmentation model on the pseudo-labelled nuScenes images

It might be quite doable since there is minimal domain gap between nuImages and nuScenes :crossed_fingers: