COCO evaluation: mAP quite low if random 1000 images are picked

[kolingv]

Refer to keras_center/bin/ctdet_coco.py (line 58), I have made the following change to randomly pick only 1000 samples :

import random
  ...
fns = sorted(glob(os.path.join(args.data, '*.jpg')))
ranom.shuffle(fns)
fns = fns[:1000]
....

bash command:

python keras_centernet/bin/ctdet_coco.py --data val2017 --annotations annotations --inres 384,384 --no-full-resolution

After evaluation,

  Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.075

which is quite low compare to the all-image evaluation, AP = 0.364

As I have done this to mask-rcnn keras, the max accuracy drop is around 2%. Here, the result dropps too much!!! Can someone explain anything wrong with the change? BTW: I have tried more samples, it seems the more images, the AP higher

[see--]

The recall is probably quite low as you just predict for 1000 images. You actually truncate the evaluation summary so we can't see it.

To get the score from these 1000 images you have to remove boxes not from these images in the ground-truth.

[kolingv]

Many thanks!! Both the AP and recall are low. And i check coco evaluation codes from ‘keras-mask-rcnn', it did remove the gound truth. I'll give a try if possilbe. Whole coco results:

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.076
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.112
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.080
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.036
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.083
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.113
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.060
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.092
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.096
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.049
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.105
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.136