search

open-mmlab / OpenPCDet

OpenPCDet Toolbox for LiDAR-based 3D Object Detection.
Apache License 2.0
4.62k stars 1.29k forks source link

Poor performance, testing on waymo dataset using kitti pretrained PointRCNN model #599

Closed julianleelinker closed 2 years ago

julianleelinker commented 3 years ago

Hi, thanks for sharing your nice work. I just try testing on waymo dataset using kitti pretrained PointRCNN model and got poor result.

Change I made in waymo_dataset.yaml:

POINT_FEATURE_ENCODING: {
    encoding_type: absolute_coordinates_encoding,
    used_feature_list: ['x', 'y', 'z', 'intensity'],
    src_feature_list: ['x', 'y', 'z', 'intensity'],
}

My pointrcnn.yaml is copied from kitti_models/pointrcnn.yaml with modification of vehicle mean_size:

POINT_HEAD:
        NAME: PointHeadBox
        CLS_FC: [256, 256]
        REG_FC: [256, 256]
        CLASS_AGNOSTIC: False
        USE_POINT_FEATURES_BEFORE_FUSION: False
        TARGET_CONFIG:
            GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
            BOX_CODER: PointResidualCoder
            BOX_CODER_CONFIG: {
                'use_mean_size': True,
                'mean_size': [
                    [4.1, 2.1, 1.7],
                    [0.8, 0.6, 1.73],
                    [1.76, 0.6, 1.73]
                ]
            }

My result:

2021-07-23 04:25:49,945   INFO  *************** EPOCH 7870 EVALUATION *****************
2021-07-23 04:26:26,205   INFO  *************** Performance of EPOCH 7870 *****************
2021-07-23 04:26:26,205   INFO  Generate label finished(sec_per_example: 0.0509 second).
2021-07-23 04:26:26,205   INFO  recall_roi_0.3: 0.086557
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.3: 0.086850
2021-07-23 04:26:26,206   INFO  recall_roi_0.5: 0.063807
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.5: 0.067835
2021-07-23 04:26:26,206   INFO  recall_roi_0.7: 0.024971
2021-07-23 04:26:26,206   INFO  recall_rcnn_0.7: 0.037225
2021-07-23 04:26:26,207   INFO  Average predicted number of objects(713 samples): 4.381
2021-07-23 04:26:47,717   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.0498
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.0468
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.0336
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.0319
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.0103
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.0077
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.0097
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.0072
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.0017
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.0016

I encountered some dependency issue so I installed waymo-open-dataset-tf-2-1-0==1.2.0, instead of waymo-open-dataset-tf-2-0 -0==1.2.0 I am not sure if that is the cause of the poor result (it seems to me not). Do you have any idea what leads to the poor result? Thank you very much.

MartinHahner commented 3 years ago

LiDAR models do not transfer well from one dataset to another one. Here is a good read from @ltriess giving an overview on the topic.

julianleelinker commented 3 years ago

Hi @MartinHahner, thanks for your reply, I understand your point. However the performance shouldn't be so bad, according to this article Train in Germany, Test in The USA: Making 3D Object Detectors Generalize , where the KITTI pretrained pointrcnn detector still gives AP(BEV) > 20% and AP(3D) > 10%. So I am wondering if I missed something in the config file or threre is some bug in the code such that models between datasets are not compatible.

MartinHahner commented 3 years ago

Train in Germany, Test in The USA: Making 3D Object Detectors Generalize

Yeah, I know that paper. It's great work. Note though, they report results using PointRCNN and not PV-RCNN. Even though PV-RCNN is superior on KITTI, it does not mean that it also transfers better to Waymo. Were you able to reproduce their results for PointRCNN?

julianleelinker commented 3 years ago

Sorry I made typos, I was actually testing using PointRCNN the result is also PointRCNN result.

MartinHahner commented 3 years ago

OK, now you have a valid point. Maybe reach out to the authors or refer to their published code on how to train on KITTI and test on Waymo.

Shuntw6096 commented 3 years ago

Hi, I have ran kitti -> waymo experiment of Train in Germany, Test in The USA: Making 3D Object Detectors Generalize, I find that a setting of pointrcnn is different from OpenPCDet and paper's code. https://github.com/cxy1997/3D_adapt_auto_driving/blob/master/pointrcnn/tools/cfgs/default.yaml#L28 So, I change kitti dataset setting https://github.com/open-mmlab/OpenPCDet/blob/c9d31d393acaae34d74cb03bd6ccff9976d3d1f3/tools/cfgs/dataset_configs/kitti_dataset.yaml#L49-L50 to

    used_feature_list: ['x', 'y', 'z',],
    src_feature_list: ['x', 'y', 'z',],

and use paper's code to translate waymo dataset to kitti format, and get a different result, perhaps the result is more reasonable, but has a gap with paper

2021-10-03 18:58:46,535   INFO  *************** EPOCH 71 EVALUATION *****************
2021-10-03 19:04:41,980   INFO  *************** Performance of EPOCH 71 *****************
2021-10-03 19:04:41,980   INFO  Generate label finished(sec_per_example: 0.1394 second).
2021-10-03 19:04:41,980   INFO  recall_roi_0.3: 0.592279
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.3: 0.597099
2021-10-03 19:04:41,981   INFO  recall_roi_0.5: 0.489929
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.5: 0.538952
2021-10-03 19:04:41,981   INFO  recall_roi_0.7: 0.130628
2021-10-03 19:04:41,981   INFO  recall_rcnn_0.7: 0.286089
2021-10-03 19:04:41,982   INFO  Average predicted number of objects(2549 samples): 6.011
2021-10-03 19:04:46,425   INFO  Car AP@0.70, 0.70, 0.70:
bbox AP:36.8763, 38.8078, 36.5180
bev  AP:39.4154, 41.3404, 37.1790
3d   AP:11.1114, 11.4454, 12.1022
aos  AP:36.64, 38.55, 36.30
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:33.8597, 35.4379, 35.6711
bev  AP:37.5705, 38.7479, 36.7095
3d   AP:9.0654, 10.1663, 10.1325
aos  AP:33.62, 35.19, 35.37
Car AP@0.70, 0.50, 0.50:
bbox AP:36.8763, 38.8078, 36.5180
bev  AP:70.8551, 70.4873, 69.7542
3d   AP:65.7294, 61.2301, 60.8189
aos  AP:36.64, 38.55, 36.30
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:33.8597, 35.4379, 35.6711
bev  AP:73.8100, 71.9384, 67.7279
3d   AP:64.7294, 63.3294, 59.4869
aos  AP:33.62, 35.19, 35.37

and run stat norm code between kitti and waymo, re-train pointrcnn with kitti after stat norm, the result is similar to the paper, but also has a gap with paper.

2021-10-04 17:07:55,597   INFO  *************** EPOCH 80 EVALUATION *****************
2021-10-04 17:14:03,482   INFO  *************** Performance of EPOCH 80 *****************
2021-10-04 17:14:03,483   INFO  Generate label finished(sec_per_example: 0.1443 second).
2021-10-04 17:14:03,483   INFO  recall_roi_0.3: 0.626237
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.3: 0.628432
2021-10-04 17:14:03,483   INFO  recall_roi_0.5: 0.581303
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.5: 0.593398
2021-10-04 17:14:03,483   INFO  recall_roi_0.7: 0.415340
2021-10-04 17:14:03,483   INFO  recall_rcnn_0.7: 0.479685
2021-10-04 17:14:03,484   INFO  Average predicted number of objects(2549 samples): 5.109
2021-10-04 17:14:07,974   INFO  Car AP@0.70, 0.70, 0.70:
bbox AP:51.5635, 53.4044, 48.2356
bev  AP:67.8604, 61.6724, 61.0304
3d   AP:47.4996, 48.2412, 47.3004
aos  AP:51.19, 52.99, 47.88
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:50.4320, 50.7123, 47.8743
bev  AP:65.9226, 64.2689, 59.9811
3d   AP:47.9117, 48.7304, 43.9152
aos  AP:50.05, 50.33, 47.46
Car AP@0.70, 0.50, 0.50:
bbox AP:51.5635, 53.4044, 48.2356
bev  AP:71.0432, 70.7979, 62.9648
3d   AP:70.0923, 70.1391, 62.5190
aos  AP:51.19, 52.99, 47.88
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:50.4320, 50.7123, 47.8743
bev  AP:70.8459, 68.6688, 64.0247
3d   AP:70.0051, 67.9982, 63.4958
aos  AP:50.05, 50.33, 47.46

By the way, I check waymo dataset after tranlated to kitti foramt with kitti_object_vis , but it seems that the location of the boxes is somewhat deviated, not sure whether this will affect the results. Imgur Imgur

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity.

Galaxy-ZRX commented 2 years ago

Hi @Shuntw6096 , may I ask how you change the kitti_dataset.yaml as you mentioned? When I tried to delete the 'intensity', it will appear errors when loading the pretrained model pth file. Can you help give me some hint? Thanks in advance!

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 2 years ago

This issue was closed because it has been inactive for 14 days since being marked as stale.