Inquiry _ Kitti Evaluation Script

[benn94]

Could you help explaining what the result values got from "evaluate_object.cpp" is? It's said to be 41 recall discretization steps but I what I got (example in pedestrian_orientation.txt) is a 40 x 4 matrix :

0.000000 1.000000 1.000000 1.000000
0.025000 0.000000 0.000000 0.000000
0.050000 0.000000 0.000000 0.000000
0.075000 0.000000 0.000000 0.000000
0.100000 0.000000 0.000000 0.000000
0.125000 0.000000 0.000000 0.000000
0.150000 0.000000 0.000000 0.000000
0.175000 0.000000 0.000000 0.000000
0.200000 0.000000 0.000000 0.000000
0.225000 0.000000 0.000000 0.000000
0.250000 0.000000 0.000000 0.000000
0.275000 0.000000 0.000000 0.000000
0.300000 0.000000 0.000000 0.000000
0.325000 0.000000 0.000000 0.000000
0.350000 0.000000 0.000000 0.000000
0.375000 0.000000 0.000000 0.000000
0.400000 0.000000 0.000000 0.000000
0.425000 0.000000 0.000000 0.000000
0.450000 0.000000 0.000000 0.000000
0.475000 0.000000 0.000000 0.000000
0.500000 0.000000 0.000000 0.000000
0.525000 0.000000 0.000000 0.000000
0.550000 0.000000 0.000000 0.000000
0.575000 0.000000 0.000000 0.000000
0.600000 0.000000 0.000000 0.000000
0.625000 0.000000 0.000000 0.000000
0.650000 0.000000 0.000000 0.000000
0.675000 0.000000 0.000000 0.000000
0.700000 0.000000 0.000000 0.000000
0.725000 0.000000 0.000000 0.000000
0.750000 0.000000 0.000000 0.000000
0.775000 0.000000 0.000000 0.000000
0.800000 0.000000 0.000000 0.000000
0.825000 0.000000 0.000000 0.000000
0.850000 0.000000 0.000000 0.000000
0.875000 0.000000 0.000000 0.000000
0.900000 0.000000 0.000000 0.000000
0.925000 0.000000 0.000000 0.000000
0.950000 0.000000 0.000000 0.000000
0.975000 0.000000 0.000000 0.000000
1.000000 0.000000 0.000000 0.000000

I don't understand this result at all. And to calculate the avg precision, it's mentioned to do the following :

sum = 0;
for (i=0; i<=40; i+=4)
  sum += vals[i];
average = sum/11.0;

I don't know what value I should sum. Please help enlightening me.

[cguindel]

Hi @benn94,

Those .txt files were originally generated by the KITTI evaluation script: I have not modified their structure at all. The first column in the 41 x 4 matrix that you are seeing contains the recall value for every step (as you said, 40 discretization steps are used). The three remaining columns represent the precision values, in the case of the detection files, or the orientation similarity values, in the orientation files, for the different recall vales. There is one column for each of the difficulty levels in the KITTI dataset: Easy, Moderate and Hard, in this order. Please check the KITTI website for the definition of the difficulty levels.

To obtain the AP and AOS values, I encourage you to check the parser.py file in this repository, which is meant to do, basically, what you are looking for.

Please let me know if you need further help!

[benn94]

@cguindel Thank you for your explanation. I will look that up.