search

nutonomy / nuscenes-devkit

The devkit of the nuScenes dataset.
https://www.nuScenes.org
Other
2.24k stars 617 forks source link

Using mini dataset for evaluation #942

Closed ahmedius2 closed 1 year ago

ahmedius2 commented 1 year ago

Hello,

I am working on a project that requires processing lidar samples on an embedded system and calculating accuracy (mAP and NDS), which takes a very long time if the actual evaluation set that has more than 6000 samples is used. As an alternative, I am using the mini dataset to get the results quicker. Are the detection and tracking scores calculated for the mini dataset reliable? Is there a lower limit of samples to calculate these?

whyekit-motional commented 1 year ago

@ahmedius2 there are only about 400 samples in the mini split, so I don't think you would get a good representation of your performance, if you evaluated on the the mini split. I would advise you to use the validation split, as far as possible, for evaluation.