Teoge / DMPR-PS

DMPR-PS: A Novel Approach for Parking-Slot Detection Using Directional Marking-Point Regression
GNU General Public License v3.0
209 stars 48 forks source link

Incorrect Precision and Recall Metrics #38

Open YoungseokOh opened 2 months ago

YoungseokOh commented 2 months ago

Hi,

I believe there's an issue with the get_confidence_list() function.

When I used your pre-trained model, I couldn't achieve the same performance metrics as you reported.

I think the else clause should be removed because it calculates values that should not be considered.

The function should only handle cases where the prediction is a true positive and matches well with the ground truth (GT).

If a prediction does not match well with a ground truth, it should not append 0 to the same list (true_positive_list).

Appending 0 incorrectly includes non-matching predictions in the calculation, which creates problems when calculating precision and recall.

I will wait your reply.

Thanks

Teoge commented 2 months ago

For each ground truth, the algorithm attempts to find a matching prediction. If no prediction matches the ground truth (indicating that the model failed to detect the object), the algorithm appends a 0 to the true_positive_list.

When calculating precision and recall, a threshold is used to divide the true_positive_list into true positives and false negatives:

false_negatives = bisect.bisect_left(true_positive_list, thresh)
true_positives = len(true_positive_list) - false_negatives

As long as the threshold is greater than 0, these appended 0s in the true_positive_list will always be treated as false negatives, which is the correct behavior. Ignoring them would result in an incorrect false negative count, leading to underestimation.

ihoofans commented 5 days ago

for i in range(num_samples): ground_truths = ground_truths_list[i] predictions = predictions_list[i] prediction_matched = [False] * len(predictions) for ground_truth in ground_truths: idx = match_gt_with_preds(ground_truth, predictions, match_labels) if idx >= 0: prediction_matched[idx] = True true_positive_list.append(predictions[idx][0]) else: true_positive_list.append(.0) for idx, pred_matched in enumerate(prediction_matched): if not pred_matched: false_positive_list.append(predictions[idx][0]) 这样有些奇怪,个人的看法应该是对于某个预测值,用真值来判断,而不是对于每个真值来用预测值来匹配,true_positive_list应该是预测值的长度,后面根据置信度阈值计算rank,可能是我不太成熟的想法,或者没有理解代码,希望作者回复,谢谢!