How to evaluate model with different maxdets in mmdet3.x?

[Artificial-Inability]

In my custom dataset, there are many objects in a single image. Therefore the default maxdets=100 is too small to evaluate the ability of a model. I tried to set the 'proposal' atribute in CocoMetric but it still display the mAP of maxdet=100. How could I set maxdet to 300?

[BIGWangYuDong]

https://github.com/open-mmlab/mmdetection/blob/f78af7785ada87f1ced75a2313746e4ba3149760/mmdet/evaluation/metrics/coco_metric.py#L39-L40

[jacksonxu]

https://github.com/open-mmlab/mmdetection/blob/f78af7785ada87f1ced75a2313746e4ba3149760/mmdet/evaluation/metrics/coco_metric.py#L39-L40

When I say 'proposal' atribute, I actually mean 'proposal_nums', I set it but it doesn't work. I will describe it in detail.

1. Default setting When I use default settings like this: The output is:

It seems correct but I found that the first line print '...maxDets=100 ] = 26.7' while later the mAP line prints '| pig | 0.281'. The test dataset is some photos from a pig farm with hundreds of pigs in a single image, I assume the inconsistency is caused by maxDet setting (The later 28.1 is the mAP w.r.t maxDets > 100, probably all the boxes in the model outputs). Then I tried to set the 'proposal_nums'.

2. Set proposal_nums to (300,) My setting was: However, it comes to an error message, there is a hard-coded 'self.params.maxDets[2]' in pycocotools

3. Set proposal_nums to (300, 300, 300) Then, I set it to (300, 300, 300) like this: The result is:

It seems good to me for the data after the first line(maxDets=300). However, the most important mAP data still prints(maxDets=100) with a '-1.000' result. So how could I get the AP of maxDets=300? Thanks!

[Artificial-Inability]

https://github.com/open-mmlab/mmdetection/blob/f78af7785ada87f1ced75a2313746e4ba3149760/mmdet/evaluation/metrics/coco_metric.py#L39-L40

When I say 'proposal' atribute, I actually mean 'proposal_nums', I set it but it doesn't work. I will describe it in detail.

1. Default setting When I use default settings like this: The output is:

It seems correct but I found that the first line print '...maxDets=100 ] = 26.7' while later the mAP line prints '| pig | 0.281'. The test dataset is some photos from a pig farm with hundreds of pigs in a single image, I assume the inconsistency is caused by maxDet setting (The later 28.1 is the mAP w.r.t maxDets > 100, probably all the boxes in the model outputs). Then I tried to set the 'proposal_nums'.

2. Set proposal_nums to (300,) My setting was: However, it comes to an error message, there is a hard-coded 'self.params.maxDets[2]' in pycocotools
3. Set proposal_nums to (300, 300, 300) Then, I set it to (300, 300, 300) like this: The result is:

It seems good to me for the data after the first line(maxDets=300). However, the most important mAP data still prints(maxDets=100) with a '-1.000' result. So how could I get the AP of maxDets=300? Thanks!

BTW, I forgot to switch my account on this computer. Feel free to reply.

[collinmccarthy]

In case anyone else stumbles upon this, the "issue" is the first mAP is hard-coded to maxDets=100, implicitly, in the COCO API. See https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L460.

This is also discussed in https://github.com/cocodataset/cocoapi/issues/558