FPS in result

[Yunge6666]

I'm confused about obtaining FPS for nuScenes dataset. In my results I got task/s and eval time. But I don’t understand what those represent?

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 5.4 task/s, elapsed: 15s, ETA: 0sStep1 Done. Start to convert detection format... [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 81/81, 86.0 task/s, elapsed: 1s, ETA: 0s Results writes to /tmp/tmpwb0m71nk/results/results_nusc.json mAP: 0.5716 mATE: 0.4096 mASE: 0.4423 mAOE: 0.4780 mAVE: 0.4115 mAAE: 0.3107 NDS: 0.5806 Eval time: 1.2s

Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.925 0.170 0.156 0.101 0.119 0.070 truck 0.804 0.157 0.127 0.068 0.071 0.000 bus 0.994 0.173 0.076 0.021 0.556 0.302 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.933 0.125 0.254 0.331 0.215 0.114 motorcycle 0.736 0.191 0.257 0.400 0.053 0.000 bicycle 0.589 0.198 0.227 0.381 0.278 0.000 traffic_cone 0.734 0.082 0.325 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan

[whyekit-motional]

@Yunge6666 the timings you see when you run the nuScenes evaluation code has nothing to do with FPS :smile: FPS should be measured by you during inference (the nuScenes evaluation code only operates on the predictions produced after inference)

[Yunge6666]

@Yunge6666 the timings you see when you run the nuScenes evaluation code has nothing to do with FPS 😄 FPS should be measured by you during inference (the nuScenes evaluation code only operates on the predictions produced after inference)

Thank you very much for your reply. Do you know what the task/s above means?

[Yunge6666]

In addition, the mini dataset I used, the 81 above is also confusing to me. Because I input camera pictures from 6 views and the corresponding lidar information. So 81 represents 81 camera images or 81 sets of inputs (camera pictures from 6 perspectives and corresponding lidar information)?

[whyekit-motional]

The 81 probably represents the number of samples in the val set of the mini split:

from nuscenes.nuscenes import NuScenes
from nuscenes.utils.splits import mini_val

nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=False)

num_samples_in_split = 0
for scene in nusc.scene:
    if scene['name'] in mini_val:
        num_samples_in_split += scene['nbr_samples']

print(f'There are {num_samples_in_split} samples.')