search

thodan / bop_toolkit

A Python toolkit of the BOP benchmark for 6D object pose estimation.
http://bop.felk.cvut.cz
MIT License
376 stars 135 forks source link

Support ADD-(S) AUC metric? #37

Closed hiyyg closed 3 years ago

hiyyg commented 4 years ago

Some methods like PoseCNN evaluate the Area Under Curve of ADD-(S), could bop_toolkit also support this feature?

thodan commented 3 years ago

This feature is already supported. The Area Under Curve is given by the average of the ADD-(S) value for different settings of the threshold of correctness (please correct me if I am wrong). You can calculate it by adding the following code to the 'errors' dictionary in https://github.com/thodan/bop_toolkit/blob/master/scripts/eval_bop19.py. 'correct_th' is a list of the thresholds of correctness for which the ADD-(S) value is calculated and averaged over.

{ 'n_top': -1, 'type': 'ad', 'correct_th': [[th] for th in np.arange(0.05, 0.51, 0.05)] }

hiyyg commented 3 years ago

Thanks. But in PoseCNN the AUC is calculated with a absolute maximum threshold 10cm, the "ad" in this toolkit is normalized by diameter. So changes are still needed.

thodan commented 3 years ago

You can remove 'ad' from the 'normalized_by_diameter' list if you don't want to normalize the error. You can find the list on this line: https://github.com/thodan/bop_toolkit/blob/master/scripts/eval_calc_scores.py#L54