How to Interpret the Classification Report

[colinyong]

Hello, I use Simba to train different classifiers and output classification reports. For some classifiers, the precision is very high, but the recall value is very low. 1）I'm wondering how to handle this situation, is it ok if I only focus on precision? 2）By setting hyperparameters, the recall value can be improved, but at the same time the precision value will decrease. To what extent do I need to adjust the hyperparameters? Is it when f1 is the biggest?

[sronilsson]

Hi @colinyong !

1. If you only focus on precision, you are likely to get a lot of false negatives: i.e., you will incorrectly classify frames as not containing behavior when then in fact do contain the behavior. You’d want to have a good recall too.
2. Hyper-parameters can be helpful, but the thing that helps the very most is re-sampling: I don’t know how many annotations for behavior present and behavior absent you have, but you want to titrate the balance between them that you use to train the classifiers. Have you used random under sampling as documented here next to the Under sample setting bullet point? I recommend setting the Under sample setting to random undersample and the under sample ratio to 1.0 to see if that helps.

[colinyong]

Thank you very much. In fact, I annotated at least ten thousand frames for each behavior from different videos and also set the Under sample settings to modify the classifiers, but the result was still the same. Next, I plan to set Over sample settings to see if it can improve the performance of classifier.

[sronilsson]

Got it! Another way to understand more what's going on: If you look at the classification videos, are there anything obvious in common you can see with the frames the classifier is missing.