IoU & confidence threshold results

[stark-t]

I did some testing on the IoU and confidence thresholds according to the commet in issue.

• [x] Grid results for YOLOv4
• [x] Grid results for YOLOv5
• [x] Grid results for YOLOv7

[valentinitnelav]

This looks fantastic Tommi! I presume you would need from me detection folders with annotation txt files for YOLOv4 for an array of IoUs and Conf values, no? I managed to run yesterday a loop on the cluster for yolov7 as well. I will set one for yolov4. Since it takes only a few minutes per iteration, I would expand the search grid in the 0.1 - 1.0 interval with a step of 0.1.

[stark-t]

@valentinitnelav ok perfect. for a complete test .1-1. would be beneficial, but I think the max values are probably below .5

[stark-t]

[stark-t]

[stark-t]

[stark-t]

[stark-t]

	IoU	OA	@IoU Threshold	@Conf Threshold
YOLOv4	.6738	.2830	.1	.2
YOLOv4	.6781	.3909	.1	.9
YOLOv5n	.6800	.8973	.2	.9
YOLOv5s	.7265	.9110	.2	.4
YOLOv5s	.6900	.9394	.2	.9
YOLOv7	.7828	.9553	.3	.8
YOLOv7	.7748	.9589	.3	.9

[stark-t]

@valentinitnelav now that I check the results I think i have mixed the axis between IoU threshold and Confidence threshold. I have to check this again

[valentinitnelav]

Thanks Tommi, I am surprised by how badly YOLOv4 performs, but that makes for a nice discussion paragraph(s) I presume.

I do not know how to interpret the negative values in the IoU graphs. I was expecting online values > 0

Also, now I realize that we should also include the rate of false positives somehow. I think the predictions set at small values for IoU & conf generate way too many boxes, or do you consider only the one(s) with the highest confidence/probability from the label files?

Indeed, would be great if you can have both axes from 0.1 to 0.9 with the origin in the left bottom corner.

[stark-t]

@valentinitnelav I just wrote you an email with the update 9*9 threshold results.

• The results include the thresholds for the IoU, overall accuracy and the flase positive rate (FPR) (there is a spelling mistake in the filenames TNR should be names FPR!)

• IoUs are only negative if there is not prediction bounding box to a label bounding box or vice versa. If there are two bounding box that don't overlap then the IoU is 0.

[stark-t]

Update best thresholds scores

IoU and OA are max scores False positive rate FPR is min score

		best score	@conf	@IoU
	IOU	.6781	.1	.9
YOLOv4	OA	.3909	.1	.9
	FPR	.0746	.1	.9
	IOU	.6800	.2	.9
YOLOv5n	OA	.8900	.1	.9
	FPR	.0132	.2	.9
	IOU	.7265	.2	.4
YOLOv5s	OA	.9339	.3	.9
	FPR	.0077	.2	.9
	IOU	.7815	.4	.9
YOLOv7	OA	.9589	.4	.9
	FPR	.0050	.3	.9

@valentinitnelav I haven't looked to deep into the IoU threshold. my fear is that using a high threshold somehow changes the actual number of items in the confusion matrix. I have to double check this effect.

[valentinitnelav]

See this interesting comment where Alexey Bochkovskiy (one of the co-developers of YOLOv7) compares YOLOv7 with several older versions https://github.com/WongKinYiu/yolov7/issues/429#issuecomment-1207070671

Seeing that comment makes me think that running tests with different values for IoU and conf was a very good idea. Thank you for helping with implementing it.