[Docs] The results of validation during training are inconsistent with the results of validation using test.py on ScanNet v2 dataset

Branch

main branch https://mmdetection3d.readthedocs.io/en/latest/

📚 The doc issue

For the environment, I used mmdet3d==1.0.0rc3, mmcv-full==1.5.0, mmdet==2.24.1, mmsegmentation==0.30.0, torch==1.8.1 During training, I used the command as follows: python tools/train.py configs/votenet/votenet_8x8_scannet-3d-18class.py Then I get the results for the 1 epoch: +----------------+---------+---------+---------+---------+ | classes | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 | +----------------+---------+---------+---------+---------+ | cabinet | 0.0709 | 0.6102 | 0.0047 | 0.1263 | | bed | 0.7858 | 0.9383 | 0.1531 | 0.4321 | | chair | 0.6624 | 0.8165 | 0.1696 | 0.3874 | | sofa | 0.5704 | 0.9072 | 0.2992 | 0.5773 | | table | 0.3508 | 0.7829 | 0.0495 | 0.2600 | | door | 0.0898 | 0.6124 | 0.0092 | 0.1242 | | window | 0.0152 | 0.4043 | 0.0001 | 0.0248 | | bookshelf | 0.1612 | 0.8701 | 0.0019 | 0.1169 | | picture | 0.0005 | 0.0676 | 0.0000 | 0.0045 | | counter | 0.0749 | 0.4615 | 0.0000 | 0.0192 | | desk | 0.4886 | 0.9291 | 0.0641 | 0.4016 | | curtain | 0.0735 | 0.6119 | 0.0000 | 0.0000 | | refrigerator | 0.0906 | 0.9123 | 0.0051 | 0.1930 | | showercurtrain | 0.1649 | 0.7857 | 0.0098 | 0.1429 | | toilet | 0.7435 | 0.9655 | 0.3817 | 0.6034 | | sink | 0.0890 | 0.4286 | 0.0001 | 0.0204 | | bathtub | 0.2969 | 0.6129 | 0.0032 | 0.0323 | | garbagebin | 0.0742 | 0.4642 | 0.0011 | 0.0679 | +----------------+---------+---------+---------+---------+ | Overall | 0.2668 | 0.6767 | 0.0640 | 0.1963 | +----------------+---------+---------+---------+---------+ When I used test.py and the weight of the 1 epoch to validate, I used the command as follows: python tools/test.py configs/votenet/votenet_8x8_scannet-3d-18class.py work_dirs/votenet_8x8_scannet-3d-18class/epoch_1.pth --eval mAP Then I get the results for the 1 epoch: +----------------+---------+---------+---------+---------+ | classes | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 | +----------------+---------+---------+---------+---------+ | cabinet | 0.0794 | 0.6344 | 0.0032 | 0.1075 | | bed | 0.7322 | 0.9383 | 0.1411 | 0.3951 | | chair | 0.6614 | 0.8173 | 0.1636 | 0.3874 | | sofa | 0.4917 | 0.9278 | 0.2106 | 0.4948 | | table | 0.3441 | 0.8057 | 0.0692 | 0.2686 | | door | 0.0869 | 0.6210 | 0.0070 | 0.1478 | | window | 0.0215 | 0.4539 | 0.0000 | 0.0284 | | bookshelf | 0.1492 | 0.8701 | 0.0035 | 0.1558 | | picture | 0.0004 | 0.0631 | 0.0002 | 0.0045 | | counter | 0.0568 | 0.4231 | 0.0002 | 0.0577 | | desk | 0.4855 | 0.9291 | 0.0525 | 0.3543 | | curtain | 0.0505 | 0.6716 | 0.0000 | 0.0149 | | refrigerator | 0.0709 | 0.8947 | 0.0003 | 0.1404 | | showercurtrain | 0.1829 | 0.8571 | 0.0015 | 0.0714 | | toilet | 0.7080 | 0.9483 | 0.3943 | 0.6552 | | sink | 0.1596 | 0.5306 | 0.0080 | 0.0510 | | bathtub | 0.2003 | 0.4839 | 0.0323 | 0.0323 | | garbagebin | 0.0760 | 0.4830 | 0.0015 | 0.0868 | +----------------+---------+---------+---------+---------+ | Overall | 0.2532 | 0.6863 | 0.0605 | 0.1919 | +----------------+---------+---------+---------+---------+

From the above results, they are not the same. When I used other weights to validate or other version of mmdet3d to train and validate, this issue still occurs. Where did this problem occur and how should I solve it? Hope to get your reply, thank you!

Suggest a potential alternative/fix

No response

open-mmlab / mmdetection3d