Request for per-category scores of BEVFusion on NUscences Val

[FrontierBreaker]

Could you provide the per-category scores of image-only/lidar-only/fusion-based BEVFusion models (the models stated in the README)? I would like to have a look on that but I could not have the resources to make a reproduction. Thanks!

[kentang-mit]

Yes. The results are given below:

[kentang-mit]

BEVFusion for detection:

Object Class    AP  ATE ASE AOE AVE AAE
car 0.892   0.170   0.148   0.060   0.273   0.185
truck   0.645   0.328   0.181   0.092   0.245   0.217
bus 0.754   0.337   0.189   0.063   0.435   0.276
trailer 0.427   0.521   0.202   0.625   0.206   0.141
construction_vehicle    0.303   0.730   0.432   0.808   0.117   0.295
pedestrian  0.881   0.134   0.288   0.388   0.218   0.100
motorcycle  0.785   0.185   0.249   0.218   0.354   0.271
bicycle 0.649   0.168   0.256   0.431   0.194   0.014
traffic_cone    0.795   0.122   0.317   nan nan nan
barrier 0.720   0.179   0.277   0.054   nan nan

BEVFusion for map segmentation (see map/[class name]iou@max for the per-class numbers):

{'map/drivable_area/iou@max': 0.8534902930259705, 'map/drivable_area/iou@0.35': 0.8425150513648987, 'map/drivable_area/iou@0.40': 0.850277304649353, 'map/drivable_area/iou@0.45': 0.8534902930259705, 'map/drivable_area/iou@0.50': 0.8519654273986816, 'map/drivable_area/iou@0.55': 0.8455023765563965, 'map/drivable_area/iou@0.60': 0.8340317010879517, 'map/drivable_area/iou@0.65': 0.8176557421684265, 'map/ped_crossing/iou@max': 0.6069052219390869, 'map/ped_crossing/iou@0.35': 0.5801296234130859, 'map/ped_crossing/iou@0.40': 0.598396360874176, 'map/ped_crossing/iou@0.45': 0.6069052219390869, 'map/ped_crossing/iou@0.50': 0.6050409078598022, 'map/ped_crossing/iou@0.55': 0.5924764275550842, 'map/ped_crossing/iou@0.60': 0.569712221622467, 'map/ped_crossing/iou@0.65': 0.5364736914634705, 'map/walkway/iou@max': 0.6734170317649841, 'map/walkway/iou@0.35': 0.6410953998565674, 'map/walkway/iou@0.40': 0.6633092164993286, 'map/walkway/iou@0.45': 0.6734170317649841, 'map/walkway/iou@0.50': 0.6716809272766113, 'map/walkway/iou@0.55': 0.6586301326751709, 'map/walkway/iou@0.60': 0.6356387138366699, 'map/walkway/iou@0.65': 0.6037718057632446, 'map/stop_line/iou@max': 0.5112313628196716, 'map/stop_line/iou@0.35': 0.4810919463634491, 'map/stop_line/iou@0.40': 0.5051711797714233, 'map/stop_line/iou@0.45': 0.5112313628196716, 'map/stop_line/iou@0.50': 0.49996575713157654, 'map/stop_line/iou@0.55': 0.4737917482852936, 'map/stop_line/iou@0.60': 0.4371887147426605, 'map/stop_line/iou@0.65': 0.3931584358215332, 'map/carpark_area/iou@max': 0.595729649066925, 'map/carpark_area/iou@0.35': 0.5829365253448486, 'map/carpark_area/iou@0.40': 0.5938929319381714, 'map/carpark_area/iou@0.45': 0.595729649066925, 'map/carpark_area/iou@0.50': 0.5878669023513794, 'map/carpark_area/iou@0.55': 0.5713744163513184, 'map/carpark_area/iou@0.60': 0.5478577017784119, 'map/carpark_area/iou@0.65': 0.5173169374465942, 'map/divider/iou@max': 0.5359243154525757, 'map/divider/iou@0.35': 0.5051995515823364, 'map/divider/iou@0.40': 0.5289976596832275, 'map/divider/iou@0.45': 0.5359243154525757, 'map/divider/iou@0.50': 0.5227832198143005, 'map/divider/iou@0.55': 0.4920440912246704, 'map/divider/iou@0.60': 0.44773176312446594, 'map/divider/iou@0.65': 0.3920497000217438, 'map/mean/iou@max': 0.6294496655464172}

[kentang-mit]

LiDAR-only baseline (TransFusion-L) for detection:

Object Class    AP  ATE ASE AOE AVE AAE
car 0.869   0.175   0.151   0.083   0.272   0.187
truck   0.612   0.329   0.183   0.090   0.233   0.231
bus 0.725   0.345   0.184   0.056   0.443   0.257
trailer 0.417   0.529   0.198   0.609   0.217   0.145
construction_vehicle    0.275   0.724   0.425   0.912   0.118   0.305
pedestrian  0.865   0.136   0.284   0.377   0.216   0.091
motorcycle  0.715   0.190   0.251   0.202   0.395   0.255
bicycle 0.559   0.165   0.255   0.441   0.207   0.014
traffic_cone    0.732   0.129   0.321   nan nan nan
barrier 0.698   0.187   0.278   0.059   nan nan

LiDAR-only baseline for segmentation:

{'map/drivable_area/iou@max': 0.7561378479003906, 'map/drivable_area/iou@0.35': 0.6824245452880859, 'map/drivable_area/iou@0.40': 0.7339752912521362, 'map/drivable_area/iou@0.45': 0.7541437149047852, 'map/drivable_area/iou@0.50': 0.7561378479003906, 'map/drivable_area/iou@0.55': 0.7430734038352966, 'map/drivable_area/iou@0.60': 0.7175263166427612, 'map/drivable_area/iou@0.65': 0.6819607019424438, 'map/ped_crossing/iou@max': 0.48328450322151184, 'map/ped_crossing/iou@0.35': 0.4326503574848175, 'map/ped_crossing/iou@0.40': 0.4713524580001831, 'map/ped_crossing/iou@0.45': 0.48328450322151184, 'map/ped_crossing/iou@0.50': 0.46618393063545227, 'map/ped_crossing/iou@0.55': 0.4207271337509155, 'map/ped_crossing/iou@0.60': 0.3579612970352173, 'map/ped_crossing/iou@0.65': 0.2879191040992737, 'map/walkway/iou@max': 0.5746684670448303, 'map/walkway/iou@0.35': 0.5179252624511719, 'map/walkway/iou@0.40': 0.55965656042099, 'map/walkway/iou@0.45': 0.5746684670448303, 'map/walkway/iou@0.50': 0.5623592734336853, 'map/walkway/iou@0.55': 0.5266869068145752, 'map/walkway/iou@0.60': 0.4747578203678131, 'map/walkway/iou@0.65': 0.4129181206226349, 'map/stop_line/iou@max': 0.3644048869609833, 'map/stop_line/iou@0.35': 0.34264904260635376, 'map/stop_line/iou@0.40': 0.3644048869609833, 'map/stop_line/iou@0.45': 0.3522408604621887, 'map/stop_line/iou@0.50': 0.3145027160644531, 'map/stop_line/iou@0.55': 0.2639297842979431, 'map/stop_line/iou@0.60': 0.21132594347000122, 'map/stop_line/iou@0.65': 0.1614154577255249, 'map/carpark_area/iou@max': 0.3164410889148712, 'map/carpark_area/iou@0.35': 0.3164410889148712, 'map/carpark_area/iou@0.40': 0.30536335706710815, 'map/carpark_area/iou@0.45': 0.2864561676979065, 'map/carpark_area/iou@0.50': 0.26261603832244873, 'map/carpark_area/iou@0.55': 0.23477502167224884, 'map/carpark_area/iou@0.60': 0.20451432466506958, 'map/carpark_area/iou@0.65': 0.17333467304706573, 'map/divider/iou@max': 0.4189370274543762, 'map/divider/iou@0.35': 0.38710689544677734, 'map/divider/iou@0.40': 0.4189370274543762, 'map/divider/iou@0.45': 0.41090458631515503, 'map/divider/iou@0.50': 0.36788326501846313, 'map/divider/iou@0.55': 0.3076866865158081, 'map/divider/iou@0.60': 0.2439807802438736, 'map/divider/iou@0.65': 0.18202926218509674, 'map/mean/iou@max': 0.4856456220149994}

Camera-only baseline (variant of BEVDet, 1 frame) for detection:

Object Class    AP  ATE ASE AOE AVE AAE
car 0.538   0.529   0.156   0.128   0.981   0.223
truck   0.290   0.723   0.215   0.143   0.905   0.231
bus 0.404   0.714   0.188   0.091   1.557   0.307
trailer 0.179   0.962   0.238   0.495   0.763   0.073
construction_vehicle    0.092   0.889   0.505   1.098   0.152   0.383
pedestrian  0.386   0.703   0.301   1.378   0.861   0.750
motorcycle  0.341   0.647   0.247   0.529   1.566   0.088
bicycle 0.264   0.558   0.257   1.053   0.378   0.021
traffic_cone    0.544   0.464   0.335   nan nan nan
barrier 0.517   0.489   0.285   0.137   nan nan

Camera-only baseline (variant of LSS, 1 frame) for segmentation:

{'map/drivable_area/iou@max': 0.8171885013580322, 'map/drivable_area/iou@0.35': 0.8066084384918213, 'map/drivable_area/iou@0.40': 0.8143597841262817, 'map/drivable_area/iou@0.45': 0.8171885013580322, 'map/drivable_area/iou@0.50': 0.814768373966217, 'map/drivable_area/iou@0.55': 0.8068908452987671, 'map/drivable_area/iou@0.60': 0.7937943339347839, 'map/drivable_area/iou@0.65': 0.7756903171539307, 'map/ped_crossing/iou@max': 0.5455877780914307, 'map/ped_crossing/iou@0.35': 0.525610089302063, 'map/ped_crossing/iou@0.40': 0.5399985313415527, 'map/ped_crossing/iou@0.45': 0.5455877780914307, 'map/ped_crossing/iou@0.50': 0.5418495535850525, 'map/ped_crossing/iou@0.55': 0.5286178588867188, 'map/ped_crossing/iou@0.60': 0.5061153173446655, 'map/ped_crossing/iou@0.65': 0.4752734303474426, 'map/walkway/iou@max': 0.5845987796783447, 'map/walkway/iou@0.35': 0.5578734278678894, 'map/walkway/iou@0.40': 0.576753556728363, 'map/walkway/iou@0.45': 0.5845987796783447, 'map/walkway/iou@0.50': 0.5808560848236084, 'map/walkway/iou@0.55': 0.5661828517913818, 'map/walkway/iou@0.60': 0.5427066683769226, 'map/walkway/iou@0.65': 0.5119655728340149, 'map/stop_line/iou@max': 0.47333601117134094, 'map/stop_line/iou@0.35': 0.4503317177295685, 'map/stop_line/iou@0.40': 0.4698221683502197, 'map/stop_line/iou@0.45': 0.47333601117134094, 'map/stop_line/iou@0.50': 0.460783988237381, 'map/stop_line/iou@0.55': 0.4348505735397339, 'map/stop_line/iou@0.60': 0.39968907833099365, 'map/stop_line/iou@0.65': 0.358795702457428, 'map/carpark_area/iou@max': 0.5398412346839905, 'map/carpark_area/iou@0.35': 0.5302309989929199, 'map/carpark_area/iou@0.40': 0.5389776229858398, 'map/carpark_area/iou@0.45': 0.5398412346839905, 'map/carpark_area/iou@0.50': 0.5317400693893433, 'map/carpark_area/iou@0.55': 0.5150808691978455, 'map/carpark_area/iou@0.60': 0.4908956289291382, 'map/carpark_area/iou@0.65': 0.4606020748615265, 'map/divider/iou@max': 0.46484124660491943, 'map/divider/iou@0.35': 0.4382099211215973, 'map/divider/iou@0.40': 0.45864635705947876, 'map/divider/iou@0.45': 0.46484124660491943, 'map/divider/iou@0.50': 0.4533892273902893, 'map/divider/iou@0.55': 0.42805343866348267, 'map/divider/iou@0.60': 0.39213383197784424, 'map/divider/iou@0.65': 0.3472171723842621, 'map/mean/iou@max': 0.5708989500999451}

[kentang-mit]

Closed since the issue is resolved.