Open arnaghizadeh opened 3 years ago
tooHotSpot
commented
2 years ago Same problem, did not expect to encounter a problem with a so popular package
UPD: all imgIDs, gtIDs and detIDs have to correspond to ranges defined in Params class. Solved when tried to evaluate on 1 class in dataset only.
visbec
commented
2 years ago I encountered the same problem and found the following, maybe this is relevant for you:
COCOeval returns zeros if all coordinates of the segmentation field are negative. Since negative coordinates for segmentation does not make any sense, these values are set to zero. If no object is present for a coco_size -1 is returned.
To prove this statement I build the following mock:
Used data as dummy.json:
{
"categories": [
{
"id": 0,
"name": "dummy",
"supercategory": ""
}
],
"images": [
{
"id": 1,
"width": 1920,
"height": 1080,
"file_name": "image.jpg",
"license": 0,
"flickr_url": "",
"coco_url": "",
"date_captured": 0
}
],
"annotations": [
{
"id": 1,
"image_id": 1,
"category_id": 0,
"segmentation": [
[
-1060.28,
-481.94,
-1059.11,
-514.77,
-1178.72,
-527.67,
-1183.5,
-496.0
]
],
"area": 3949.0,
"bbox": [
1059.11,
481.94,
124.39,
45.73
],
"score": 1.0,
"iscrowd": 0,
"attributes": {
"occluded": false
}
}
]
}Used script as mock:
from pycocotools.cocoeval import COCOeval
from pycocotools import coco
a = coco.COCO("dummy.json")
b = coco.COCO("dummy.json")
# coco_eval = COCOeval(coco_pred, coco_pred, 'segm')
coco_eval = COCOeval(a, b, 'segm')
coco_eval.evaluate()
coco_eval.accumulate()
coco_eval.summarize()
print(coco_eval.stats)Returns:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000If the following data is used instead (note: all segmentation coordinates are not negative):
{
"categories": [
{
"id": 0,
"name": "dummy",
"supercategory": ""
}
],
"images": [
{
"id": 1,
"width": 1920,
"height": 1080,
"file_name": "image.jpg",
"license": 0,
"flickr_url": "",
"coco_url": "",
"date_captured": 0
}
],
"annotations": [
{
"id": 1,
"image_id": 1,
"category_id": 0,
"segmentation": [
[
1060.28,
481.94,
1059.11,
514.77,
1178.72,
527.67,
1183.5,
496.0
]
],
"area": 3949.0,
"bbox": [
1059.11,
481.94,
124.39,
45.73
],
"score": 1.0,
"iscrowd": 0,
"attributes": {
"occluded": false
}
}
]
}The following ouput is generated:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 1.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 1.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000Since the bbox is in the size medium the medium metrics are one (ground truth == gold label) and the oder sizes metrics are -1.
arnaghizadeh
commented
2 years ago @visbec thanks for trying to help, but as you can see in the gt.json file, there are no negative values in the segmentation part, so I don't think it is relevant.
vkm0cherla
commented
2 years ago @arnaghizadeh this solved it for me: https://github.com/cocodataset/cocoapi/issues/507#issuecomment-857272753
arnaghizadeh
commented
2 years ago @vkm0cherla unfortunately didn't help this case.
VtojeC
commented
2 years ago Can you guys explain me what 'area' is and how is it calculated? Would be cool to know in case of evaluating bounding boxes tho.
visbec
commented
2 years ago @VtojeC Are you revering to the area of the following COCOeval print?
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000If so I may help. The area revers to the bounding box area. It is simply calculated by width times height of the bounding box. COCOeval can be configured with thresholds to define the ranges for small, medium and large. The area is used to judge the models performance dependent on the bounding box size (size==area).
VtojeC
commented
2 years ago @visbec It is too late, but I figured it myself out and posted my observations under another post with related topic. I'm sorry I didn't update the situation, but I hardly ever use github and have no idea how to get back to the threads where I posted anything.
And the following output of 0.000 or -1.000 was also normal in my case due to several mistakes I made: 1) I didn't have unique annotation ID for each annotation, 2) Information about images in annFile were repeated, I put the same filename multiple times instead of once, 3) Segmentation list was not used at all, because I was doing Detection task with BBoxes only, 4) My Results file contained sometimes a false detection in the frame that was not present in annotation file as there was nothing present in the original frame so it was not saved into the ground truth file. I added the missing image to the list of images and volia! Everything worked.
But thank you for your input. Have a nice day.
I have a strange problem with COCO eval that it returns zeros for evaluating ground truth with itself. I have a custom coco file and transformed it into the result format as shown in the pycocoEvalDemo.ipynb file. Everything should be working, and my expectation is to get 100% accuracy, but I get zeroes. Any thoughts?
The code to run the evaluation:
Results:
Here is the ground truth coco file (gt.json):
Here is the converted ground truth from standard coco file into the result coco file (gt_result.json):