Question abount mAP evaluation in article

[lostlose]

Hello, I have some question about the mAP evaluation in article. In Asynet, it uses VOC evaluation method and can only get mAP(0.5) In RED, it uses the COCO evaluation method, but does not indicate whether the result is mAP(0.5) or mAP(0.5:0.95) In your work, using the COCO evaluation method, the result is mAP (0.5:0.95) I'm really confused about how mAP is evaluated, have you figured out the above works?

[loiccordone]

Hello @lostlose,

For Asynet, you are indeed right, I wrongly reported in the article that Asynet uses mAP (0.5:0.95) but they seem to use the Pascal VOC mAP (see their reference [44]). As a guide, our VGG, DenseNet and MobileNet models respectively reaches 0.38, 0.37 and 0.35 mAP(0.5).

RED and our work both use the COCO mAP 0.5:0.95, as we both use the official COCO API. The evaluation code for Prophesee's RED are contained in their repository, while our evaluation code can be found here.

I recommend to use the official COCO API to eliminate any source of error in the calculation of metrics.

[lostlose]

Firstly, thank you very much for your answer!

In fact, I am using COCO API. But I'm wondering if RED is using mAP(0.5:0.95) because COCO API can get both mAP(0.5) and mAP(0.5:0.95). If its result is using mAP(0.5:0.95), there is a huge gap between that and the current work.

Maybe I should ask the author of RED.

Anyway, thanks again.

[loiccordone]

I am quite confident Prophesee's RED is using mAP(0.5:0.95) since in their paper they provide mAP(0.75) that are higher than their global COCO mAP.

There is obviously a huge gap between their work and ours, as we trained from scratch spiking neural networks. They use classical convolutions and LSTMs, which are much more mature.

This repository is public to encourage research in spiking neural networks for real-world applications such as object detection, and of course we are focused on closing the gap with deep neural networks!

[lostlose]

OK，thank you very much, I get it now.