low mAP and state dict loading err when testing the provided checkpoint

[richardkxu]

Hi, thanks for open sourcing the repo! I tried to test ssl-pointpillar with provided checkpoint pp_100.pth using ./tools/test_our_pointpillars_nus.sh. However, it gave me the following state dict loading warning:

load checkpoint from local path: ./checkpoints/pp_100.pth
The model and loaded state dict do not match exactly

unexpected key in source state_dict: pts_voxel_encoder.vfe_layers.1.norm.weight, pts_voxel_encoder.vfe_layers.1.norm.bias, pts_voxel_encoder.vfe_layers.1.norm.running_mean, pts_voxel_encoder.vfe_layers.1.norm.running_var, pts_voxel_encoder.vfe_layers.1.norm.num_batches_tracked, pts_voxel_encoder.vfe_layers.1.linear.weight

The test resulted in an mAP and an NDS of almost 0:

mAP | mATE | mASE | mAOE | mAVE | mAAE | NDS
0 | 0.9924 | 0.9433 | 1.0658 | 0.9776 | 0.9603 | 0.0126

Can you clarify which checkpoint and config file to use for testing different pretrained checkpoints on different dataset?

Also, can you provide checkpoints for PointPillars and CenterPoint on the kitti dataset?

[MingyuLiu1]

Hi, can you please try to use the official mmdetection3d repo to test the ckpt? Because I checked the codes and found that this config file hv_pointpillars_fpn_nus_basic.py in the training script was used for the models which only have one linear layer in the voxel_encoder, but actually, the PointPillars has two.

Sorry for the unclean codes. Recently I did not have times to update this repo. But I really need to do it.

[richardkxu]

Can you clarify which version of mmdet3d and which config file should I use? Do I still use pp_100.pth ckpt?

[HenryJunW]

Same issue, did you find out what's reason behind this? @richardkxu Thanks!

[MingyuLiu1]

Hi, please use the version of mmdet3d==0.13.0, mmcv-full==1.3.4, mmdet==2.11.0.

[HenryJunW]

Thanks for your reply. I switched from mmcv-full 1.4.0 to 1.3.4 (mmdet3d and mmdet are matched), however, the evaluation results didn't change. Am I missing anything else? Thanks!

[MingyuLiu1]

Thanks for your reply. I switched from mmcv-full 1.4.0 to 1.3.4 (mmdet3d and mmdet are matched), however, the evaluation results didn't change. Am I missing anything else? Thanks!

Hi, if the version is correct, please evaluate the checkpoint directly on the installed original mmdetection3d. You don't need to modify the mmdetection3d codes with the codes provided by us.

[HenryJunW]

Thanks again for your instant reply! I will try that!

[HenryJunW]

Hello @MingyuLiu1 , I could get the evaluated numbers matched with that in your paper using the checkpoint provided and the original mmdetection3D. However, after the model is trained by myself, the mAP is still 0 if directly switching to the original mmdetection3D during evaluation. Any comment is appreciated. Thanks!

[HenryJunW]

I’ve noticed that the checkpoint provided is using two vfe_layers, while the existing config https://github.com/emecercelik/ssl-3d-detection/blob/e605ad616278cdd7cb0c6cd5b8479c8c3921c158/mmdetection3d/mmdetection3d/configs/_base_/models/hv_pointpillars_fpn_nus_flow.py#L18 is using 1 vfe_layer. Does that make a large performance gap?

[MingyuLiu1]

I’ve noticed that the checkpoint provided is using two vfe_layers, while the existing config

https://github.com/emecercelik/ssl-3d-detection/blob/e605ad616278cdd7cb0c6cd5b8479c8c3921c158/mmdetection3d/mmdetection3d/configs/_base_/models/hv_pointpillars_fpn_nus_flow.py#L18

is using 1 vfe_layer. Does that make a large performance gap?

We did two ablation studies, one is only using one vfe_layer for both baseline and our SSL method, and the other one is leveraging two vfe_layes. For both baseline and our method, two vie_layers had around 2 points improvement than one. So the difference is obvious.