Question about the reported AP mismatch between the paper and GitHub

facebookresearch / 3detr

Code & Models for 3DETR - an End-to-end transformer model for 3D object detection

Apache License 2.0

629 stars 79 forks source link

Question about the reported AP mismatch between the paper and GitHub #30

Closed mu-cai closed 2 years ago

mu-cai commented 2 years ago

Hi, Thanks for your great work. I am just wondering the AP difference between the Scannet Results (for AP25 and AP50)

In the paper, it is 62.7 | 37.5
In Github, it is 62.1 | 37.9 here AP25 in GitHub is lower than that in the paper.

Besides, I am also confused about Scannet 720 epochs AP values: in GitHub, it is 61.1 | 40.2, where I am quite confused about the AP50 results...

imisra commented 2 years ago

Hi @mu-cai Thanks for your comments!

As noted in the README, you can expect a variance of up to 1 mAP in the model evaluation. You can see that AP25 is lower, but AP50 is higher. For ScanNet, we noticed this as well, and don't have a thorough explanation. I believe the model is overfitting, since ScanNet is a small dataset.