search

koide3 / hdl_people_tracking

Real-time people tracking using a 3D LIDAR
BSD 2-Clause "Simplified" License
288 stars 94 forks source link

Detection evaluation question #15

Closed agn-7 closed 4 years ago

agn-7 commented 5 years ago

Hi @koide3

How could I evaluate this procedure by HDL_500.bag and compute the ROC curve?

I computed the TP, TN, FP, FN, Recall, Precision, FPR, and TPR with counting the all of that sequenced data (rosbag) per each frame (summation of each TP, FP, ... per frame).

So in the end, I have an FPR and TPR, thus I cannot plot a ROC curve, because it is only a point!

What's your idea to detection evaluation (no tracking)?

koide3 commented 5 years ago

Hi @agn-7 , To plot an ROC curve, you need to test the system with varying sensitivities (i.e., detection threshold). Maybe you can change the threshold at the following line to change the detection sensitivity (I have not tried though...), and measure FPR/TPR under different thresholds to plot the curve.

https://github.com/koide3/hdl_people_tracking/blob/8c0d863e77207e7012217fcd16ce89f29a78ba67/src/kidono_human_classifier.cpp#L41https://docs.opencv.org/2.4/modules/ml/doc/boosting.html#cvboost-predict

agn-7 commented 5 years ago

Thanks for the response,

Also, we can change some parameters value in preprocessing level such as downsampling value ...

But I realized that to evaluating pedestrian detection we couldn't use ROC curve, because we cannot count the TN (non-pedestrian) in each frame. So with borrowing from this link, I found out the true evaluating for this project is only precision and recall (and F1-score to integrating them), because it is a retrieve problem and the number of TP (pedestrian) and non-pedestrian is not equal and is an imbalance.

TL;DR

When to Use ROC vs. Precision-Recall Curves?

Generally, the use of ROC curves and precision-recall curves are as follows:

  • ROC curves should be used when there are roughly equal numbers of observations for each class.
  • Precision-Recall curves should be used when there is a moderate to large class imbalance.

The reason for this recommendation is that ROC curves present an optimistic picture of the model on datasets with a class imbalance.

However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval, have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.

akamob commented 2 years ago

Hi, @agn-7,

I'm wondering if it is possible that you could share the precision-recall curve and F1-score of human detection with me? I want to quantify the performance.

Thank you!