Results interpretations

[vobecant]

Hi,

first of all let me thank you for this repo. I would like to ask you about the interpretation of results. When I run the evaluation using your DLA-34 model, I get this:

Car AP@0.70, 0.70, 0.70: bbox AP:90.84, 89.72, 80.89 bev AP:24.48, 18.83, 17.77 3d AP:17.88, 12.67, 12.00 aos AP:90.31, 88.66, 79.69 Car AP_R40@0.70, 0.70, 0.70: bbox AP:96.96, 91.32, 83.77 bev AP:23.28, 17.16, 14.86 3d AP:15.95, 11.41, 9.55 aos AP:96.34, 90.18, 82.39 Car AP@0.70, 0.50, 0.50: bbox AP:90.84, 89.72, 80.89 bev AP:61.77, 46.04, 43.18 3d AP:56.54, 43.31, 37.07 aos AP:90.31, 88.66, 79.69 Car AP_R40@0.70, 0.50, 0.50: bbox AP:96.96, 91.32, 83.77 bev AP:61.44, 45.71, 39.51 3d AP:55.12, 41.15, 35.51 aos AP:96.34, 90.18, 82.39 Pedestrian AP@0.50, 0.50, 0.50: bbox AP:69.26, 60.87, 52.07 bev AP:12.50, 11.99, 11.42 3d AP:12.28, 11.33, 10.71 aos AP:61.45, 53.35, 45.55 Pedestrian AP_R40@0.50, 0.50, 0.50: bbox AP:67.03, 57.51, 47.95 bev AP:6.74, 5.34, 4.33 3d AP:5.73, 4.27, 3.46 aos AP:59.17, 50.00, 41.50 Pedestrian AP@0.50, 0.25, 0.25: bbox AP:69.26, 60.87, 52.07 bev AP:30.74, 25.89, 20.53 3d AP:30.47, 25.37, 20.38 aos AP:61.45, 53.35, 45.55 Pedestrian AP_R40@0.50, 0.25, 0.25: bbox AP:67.03, 57.51, 47.95 bev AP:25.65, 20.78, 17.36 3d AP:25.41, 20.47, 17.11 aos AP:59.17, 50.00, 41.50 Cyclist AP@0.50, 0.50, 0.50: bbox AP:73.84, 48.44, 47.94 bev AP:15.24, 11.49, 11.31 3d AP:10.21, 6.54, 6.46 aos AP:70.27, 45.87, 45.24 Cyclist AP_R40@0.50, 0.50, 0.50: bbox AP:74.42, 46.35, 43.89 bev AP:8.87, 4.61, 4.40 3d AP:7.27, 3.70, 3.42 aos AP:70.55, 43.61, 41.22 Cyclist AP@0.50, 0.25, 0.25: bbox AP:73.84, 48.44, 47.94 bev AP:32.14, 19.41, 18.73 3d AP:29.18, 19.24, 18.56 aos AP:70.27, 45.87, 45.24 Cyclist AP_R40@0.50, 0.25, 0.25: bbox AP:74.42, 46.35, 43.89 bev AP:28.15, 15.16, 13.49 3d AP:26.90, 14.91, 13.17 aos AP:70.55, 43.61, 41.22

Can you please tell me what results correspond to? They differ from the values that you present at this page.

Thank you very much in advance.

[Banconxuan]

I have reported only the results of KM3D, which come from faster.py.

[vobecant]

Thank you for your reply. I suppose that the reported results are for the standard AP (11 points) and not for the new AP_40 with 40 points, right? Also, the training in main.py uses KM3D so it should be evaluated with faster.py? And one more question: how does it come that there are two different results for each category? E.g., two results for Car AP - 0.7, 0.7, 0.7 setup and 0.7, 0.5, 0.5 setup. I guess that those might be different setups for easy/moderate/hard, but I don't understand why would I get different result for 0.7 (first column) in both tables, or why would the first rows (2D bounding boxes) be the same. Thank you very much in advance.

[sparro12]

Upon running faster.py, we do not see any results like the ones you posted above @vobecant. Only the time stats, are printed to the terminal. Are the stats you saw being printed to a separate file and if so where is that file located within the repo? We've looked through the code, but have yet to find such a file. Thanks

[vobecant]

@sparro12, faster.py does just the detection. You then need to run src/tools/kitti-object-eval-python/evaluate.py for the evaluation.

[sparro12]

Thank you @vobecant