How to test the accuracy of my detector?

[apenemo]

Hi, I can see when I look at my annotated videos that my detector has an accuracy that could be improved. The main problem is that it tends to detect a monkey's limb as an individual wish can disturb the categorizer trainning.

So I would like to know what is the average precision of my detector, is there a way of doing that so I can check on the detectors improvements ?

Thank you very much

[yujiahu415]

Hi,

The annotated video is the most straightforward way for you to get an idea on how accurate a trained Detector is. If the monkey's limb was detected as an individual, then I think the Detector was not accurate and you probably need to add some more annotated images and train a new Detector. If the false detections of limbs was the main problem, you probably can include some frames in which this false detection occurred and annotate the images well.

If you want to obtain the value of an average precision of a Detector, you can sample certain number of frames in which the monkey has been detected (predicted positives) in an annotated video, for example, 1000 frames, and determine in each frame whether the limb was falsely considered as an individual. By this way, you can get the number of frames of true positives. Divide the number of true positive frames by the total number of predicted positives and this is the precision. LabGym does not automatically give you the precision / sensitivity of a trained Detector. This is because, to calculate the precision / sensitivity, it requires the ground truth, which can only be obtained by manual sampling / examinations of certain number of frames anyway. And this process is just the same as what I mentioned above.

[apenemo]

Thank you for your answer, I suspected that it would be the only way to know but I have to say that I hoped for a less time consuming option haha. I will do as you sugested to evaluate my detectors'acurracy. I have not enough time on my hands to train another one for now, but I will remember your wise advice for next time.

[yujiahu415]

No problem! If you have any additional issues or questions, just let me know.

[yujiahu415]

Hi, Just wanted to let you know that new functions of testing Detectors and Categorizers have been added in new version v2.2. Now you can use ground-truth datasets to test them and the testing accuracies will be printed out.