[question]About the vsd recall metric.

[zeal-up]

Hi, thanks for your work. The source code is very nicely organized! I am confused about the evaluation procedure and the recall metric. Below is my understanding: setting: 1. use ground truth bounding box for evaluation. 2. for object 1

1. extract object patch using object ground truth bounding box from all scene
2. evaluate vsd accuracy for all patch

I have some question:

1. do we use the bounding box or visible bounding box
2. I am a little confused about the recall metric in this setting(if I understand right). Since we evaluate for all object patches from all scenes and each patch only has the target object, we can calculate the accuracy but I don't know how to calculate the recall.

[MartinSmeyer]

Hi! Thanks!

1. For the evaluation with gt boxes we use the full bounding boxes encompassing also the invisible part of the object. Samples are included in evaluation when visibility is >10%.
2. Recall is portion of estimates with vsd_error < 0.3. If the same object appears multiple times in a scene, only the estimate of a single one is regarded (SISO setting).

For more details, you can check this: http://cmp.felk.cvut.cz/sixd/challenge_2017/

[zeal-up]

Thanks for your reply! I have one more question to bother you. I want to reconfirm the validation procedure for scenes with multiple instances. Do you mean that for these scenes with multiple instances, we extract multiple patches and compute the vsd_error for these patches, choose the smallest error as the error of this scene? Appreciated if you reply again.

[MartinSmeyer]

Yes, this is what the sixd toolkit for evaluation does, if I remember correctly: https://github.com/thodan/sixd_toolkit/blob/master/doc/sixd_2017_results_format.md

[zeal-up]

Thank you very much!