Low accuracy when training on waymo dataset

[LZDSJTU]

Hi, I add the recent update on the waymo dataset to the previous version of OpenPCDet (maybe June2020 version), and train second/PartA2 on the waymo dataset. I do not modify the preprocess script and model cfg for waymo.

However, the accuracy is much lower. For second, the final accuracy is as follows: 742 2020-11-24 08:00:32,763 INFO ****Evaluation done.** 743 2020-11-24 09:47:11,516 INFO EPOCH 0 EVALUATION ** 744 2020-11-24 09:48:05,627 INFO Performance of EPOCH 0 ***** 745 2020-11-24 09:48:05,627 INFO Generate label finished(sec_per_example: 0.0068 second). 746 2020-11-24 09:48:05,628 INFO recall_roi_0.3: 0.000000 (0 / 296561) 747 2020-11-24 09:48:05,628 INFO recall_rcnn_0.3: 0.689605 (204510 / 296561) 748 2020-11-24 09:48:05,628 INFO recall_roi_0.5: 0.000000 (0 / 296561) 749 2020-11-24 09:48:05,628 INFO recall_rcnn_0.5: 0.574320 (170321 / 296561) 750 2020-11-24 09:48:05,628 INFO recall_roi_0.7: 0.000000 (0 / 296561) 751 2020-11-24 09:48:05,628 INFO recall_rcnn_0.7: 0.316451 (93847 / 296561) 752 2020-11-24 09:48:05,637 INFO Average predicted number of objects(7998 samples): 100.557 753 2020-11-24 09:51:25,703 INFO 754 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.3911 755 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.3855 756 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3620 757 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3568 758 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3631 759 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2795 760 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3308 761 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2545 762 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 763 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 764 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 765 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 766 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2761 767 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2636 768 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2757 769 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2632

I'd like to ask what may cause this problem.

Thank you very much.

[LZDSJTU]

I trained 80 epoches, as the previous version does not load the "NUM_EPOCHS" parameter in the cfg

[sshaoshuai]

What's your model for evaluation? It seems you used the epoch 0 for evaulation.

[LZDSJTU]

What's your model for evaluation? It seems you used the epoch 0 for evaulation.

I used the last epoch to evaluate, which is the online evaluation during the training process. This is just a typo error.

[LZDSJTU]

And I do not use "merge_all_iters_to_one_epoch".

[sshaoshuai]

Let me check whether there is a bug in the dataloader.

[LZDSJTU]

Let me check whether there is a bug in the dataloader.

Thank you.

[sshaoshuai]

Currently I didn't find any obvious bug, maybe you could try to train with the latest codes. I will also try to train a model.

[LZDSJTU]

Currently I didn't find any obvious bug, mayby you could try to train with the latest codes. I will also try to train a model.

Okay

[sshaoshuai]

I got much better results than your reported performance, hence it should not be a problem of the dataloader. I will try more models since the released dataloader is a little bit different with me and my dataloader is an old version.

[LZDSJTU]

I just donwload the updated version and try to train it. However, out-of-memory issue happened. I do not know if the updated version need much more memory for the waymo dataset.

[sshaoshuai]

No, I could train SECOND with the given config on GPU 1080Ti with 11G memory.

[LZDSJTU]

I use the latest version. My training will always be stuck here： 2020-11-25 15:59:13,004 INFO **Start training data1/OpenPCDet-master20201117/tools/cfgs/waymo_models/second(test_in_shell)** epochs: 0%| | 0/30 [00:00<?, ?it/s]

[LZDSJTU]

It seems that the problem is caused by the multi-gpu training. When I use the single gpu, the training is OK. This problem does not happen when I use the old version. I will check it.

[LZDSJTU]

https://github.com/open-mmlab/OpenPCDet/blob/594962844729353842207b9e3c50f8b2434484f9/tools/train_utils/train_utils.py#L76

When using two gpus for training, the code is stack here. When using the old version in the same device, I think that maybe the dataloader cause the problem. Is the updated dataloader much different from the old one?

[LZDSJTU]

When I set the num_worker=0, multi-gpu training is ok. Such problem does not exsit in the previous version. I will look for the cause of this situation.

[sshaoshuai]

@LZDSJTU , I do find some problems in the evaluation codes, please see https://github.com/open-mmlab/OpenPCDet/pull/384. But the data and training codes are fine. You could follow the instructions to re-create the validation infos and re-evaluate your models.

[LZDSJTU]

OK, I will modify the evaluation code. I think that the latest version has some problem in dataloader. I run different versions in the same environment with multi-gpus, the previous version is OK but the current version is deadlock. I haven't found the reason.

[sshaoshuai]

Have you tried to run with other datasets like KITTI?

[LZDSJTU]

Yep, the similar problem still exsits for KITTI. The difference is that using KITTI may not meet deadlock. It just gets stack for a long time and start running. However, if using the previous version, it is very fast to start running.

[MartinHahner]

@sshaoshuai

I do find some problems in the evaluation codes, please see #384. But the data and training codes are fine. You could follow the instructions to re-create the validation infos and re-evaluate your models.

Saw this message just in time,minutes after my script started to create them. I had to re-download WOD (I used to work with v1.0 instead of v1.2). I will try to reproduce your results on WOD next week.

[LZDSJTU]

@LZDSJTU , I do find some problems in the evaluation codes, please see #384. But the data and training codes are fine. You could follow the instructions to re-create the validation infos and re-evaluate your models.

Hi, I re-preprocess the waymo val set and evaluate the accuracy. Actually, the accuracy increases but is still much lower than yours. I trained it with the old pcdet with the waymo cfg.

I have not trained on the latest version as the multi-gpu training has some problem in my environment before. I found that If I insert the following sentence in eval_utils.py, the multi-gpu training will be okay:

from pcdet.datasets.kitti.kitti_object_eval_python.eval import print_str

I cannot understand the reason now as the "print_str" is not used in eval_utils.py and my code is not stuck here. It got stuck at the beginning of training.

[LZDSJTU]

@sshaoshuai @MartinHahner I train the second model using the latest version, and I have re-created the val info for the waymo dataset.

I follow the instruction to train the model for 30 epochs. The final accuracy is still much lower: 5 2020-12-01 03:47:44,740 INFO * EPOCH 30 EVALUATION *** 366 2020-12-01 03:48:32,127 INFO * Performance of EPOCH 30 *** 367 2020-12-01 03:48:32,128 INFO Generate label finished(sec_per_example: 0.0059 second). 368 2020-12-01 03:48:32,128 INFO recall_roi_0.3: 0.000000 369 2020-12-01 03:48:32,128 INFO recall_rcnn_0.3: 0.668910 370 2020-12-01 03:48:32,128 INFO recall_roi_0.5: 0.000000 371 2020-12-01 03:48:32,128 INFO recall_rcnn_0.5: 0.552218 372 2020-12-01 03:48:32,128 INFO recall_roi_0.7: 0.000000 373 2020-12-01 03:48:32,128 INFO recall_rcnn_0.7: 0.297340 374 2020-12-01 03:48:32,133 INFO Average predicted number of objects(7998 samples): 101.049 375 2020-12-01 03:51:04,877 INFO 376 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4284 377 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4213 378 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3733 379 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3670 380 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4020 381 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3015 382 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3456 383 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2589 384 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 385 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 386 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 387 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 388 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2885 389 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2714 390 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2791 391 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2626

The result is very similar with that using the previous PCDet version which adds the waymo cfg (after re-creating the val info).

[LZDSJTU]

Currently I only re-create the val info for the waymo dataset. I do not know whether the train info needs to be re-created as the gt_sample needs to the parameter 'num_points_in_gt' to filter the database.

But my accuracy is very different from the your provided, so I am not sure whether the gap is caused by this reason.

[LZDSJTU]

This is my training log for SECOND model.

2020-11-30 19:51:53,592 INFO **Start logging** 2020-11-30 19:51:53,592 INFO CUDA_VISIBLE_DEVICES=ALL 2020-11-30 19:51:53,592 INFO total_batch_size: 32 2020-11-30 19:51:53,592 INFO cfg_file /data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second.yaml 2020-11-30 19:51:53,592 INFO batch_size 4 2020-11-30 19:51:53,592 INFO epochs 30 2020-11-30 19:51:53,592 INFO workers 8 2020-11-30 19:51:53,592 INFO extra_tag 20201130_second 2020-11-30 19:51:53,592 INFO ckpt None 2020-11-30 19:51:53,592 INFO pretrained_model None 2020-11-30 19:51:53,592 INFO launcher pytorch 2020-11-30 19:51:53,592 INFO tcp_port 18865 2020-11-30 19:51:53,592 INFO sync_bn False 2020-11-30 19:51:53,592 INFO fix_random_seed False 2020-11-30 19:51:53,593 INFO ckpt_save_interval 5 2020-11-30 19:51:53,593 INFO local_rank 0 2020-11-30 19:51:53,593 INFO max_ckpt_save_num 30 2020-11-30 19:51:53,593 INFO merge_all_iters_to_one_epoch False 2020-11-30 19:51:53,593 INFO set_cfgs None 2020-11-30 19:51:53,593 INFO max_waiting_mins 0 2020-11-30 19:51:53,593 INFO start_epoch 0 2020-11-30 19:51:53,593 INFO save_to_file False 2020-11-30 19:51:53,593 INFO cfg.ROOT_DIR: /data1/OpenPCDet-master_new 2020-11-30 19:51:53,593 INFO cfg.LOCAL_RANK: 0 2020-11-30 19:51:53,593 INFO cfg.CLASS_NAMES: ['Vehicle', 'Pedestrian', 'Cyclist'] 2020-11-30 19:51:53,593 INFO
cfg.DATA_CONFIG = edict() 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.DATASET: WaymoDataset 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/waymo 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.PROCESSED_DATA_TAG: waymo_processed_data 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4] 2020-11-30 19:51:53,593 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict() 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val 2020-11-30 19:51:53,593 INFO
cfg.DATA_CONFIG.SAMPLED_INTERVAL = edict() 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.SAMPLED_INTERVAL.train: 5 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.SAMPLED_INTERVAL.test: 5 2020-11-30 19:51:53,593 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict() 2020-11-30 19:51:53,593 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder'] 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['pcdet_waymo_dbinfos_train_sampled_10.pkl'], 'PREPARE': {'filter_by_min_points': ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Vehicle:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 5, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}] 2020-11-30 19:51:53,594 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict() 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity', 'elongation'] 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation'] 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 80000, 'test': 90000}}] 2020-11-30 19:51:53,594 INFO cfg.DATA_CONFIG._BASECONFIG: cfgs/dataset_configs/waymo_dataset.yaml 2020-11-30 19:51:53,594 INFO
cfg.MODEL = edict() 2020-11-30 19:51:53,594 INFO cfg.MODEL.NAME: SECONDNet 2020-11-30 19:51:53,594 INFO
cfg.MODEL.VFE = edict() 2020-11-30 19:51:53,594 INFO cfg.MODEL.VFE.NAME: MeanVFE 2020-11-30 19:51:53,595 INFO
cfg.MODEL.BACKBONE_3D = edict() 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x 2020-11-30 19:51:53,595 INFO
cfg.MODEL.MAP_TO_BEV = edict() 2020-11-30 19:51:53,595 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression 2020-11-30 19:51:53,595 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256 2020-11-30 19:51:53,595 INFO
cfg.MODEL.BACKBONE_2D = edict() 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5] 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2] 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256] 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2] 2020-11-30 19:51:53,595 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256] 2020-11-30 19:51:53,595 INFO
cfg.MODEL.DENSE_HEAD = edict() 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Vehicle', 'anchor_sizes': [[4.7, 2.1, 1.7]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.55, 'unmatched_threshold': 0.4}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.91, 0.86, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.78, 0.84, 1.78]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}] 2020-11-30 19:51:53,595 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict() 2020-11-30 19:51:53,595 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder 2020-11-30 19:51:53,596 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict() 2020-11-30 19:51:53,596 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2 2020-11-30 19:51:53,596 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 2020-11-30 19:51:53,596 INFO
cfg.MODEL.POST_PROCESSING = edict() 2020-11-30 19:51:53,596 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7] 2020-11-30 19:51:53,596 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1 2020-11-30 19:51:53,596 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False 2020-11-30 19:51:53,596 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: waymo 2020-11-30 19:51:53,596 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict() 2020-11-30 19:51:53,596 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False 2020-11-30 19:51:53,597 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu 2020-11-30 19:51:53,597 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01 2020-11-30 19:51:53,597 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096 2020-11-30 19:51:53,597 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500 2020-11-30 19:51:53,597 INFO
cfg.OPTIMIZATION = edict() 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 30 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.LR: 0.003 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85] 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.PCT_START: 0.4 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45] 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07 2020-11-30 19:51:53,597 INFO cfg.OPTIMIZATION.LR_WARMUP: False 2020-11-30 19:51:53,598 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1 2020-11-30 19:51:53,598 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10 2020-11-30 19:51:53,598 INFO cfg.TAG: second 2020-11-30 19:51:53,598 INFO cfg.EXP_GROUP_PATH: data1/OpenPCDet-master_new/tools/cfgs/waymo_models 2020-11-30 19:51:56,666 INFO Database filter by min points Vehicle: 477761 => 310377 2020-11-30 19:51:56,697 INFO Database filter by min points Pedestrian: 222675 => 136965 2020-11-30 19:51:56,700 INFO Database filter by min points Cyclist: 5344 => 3717 2020-11-30 19:51:56,794 INFO Database filter by difficulty Vehicle: 310377 => 310377 2020-11-30 19:51:56,836 INFO Database filter by difficulty Pedestrian: 136965 => 136965 2020-11-30 19:51:56,837 INFO Database filter by difficulty Cyclist: 3717 => 3717 2020-11-30 19:51:57,209 INFO Loading Waymo dataset 2020-11-30 19:52:07,651 INFO Total skipped info 0 2020-11-30 19:52:07,651 INFO Total samples for Waymo dataset: 158081 2020-11-30 19:52:07,659 INFO Total sampled samples for Waymo dataset: 31617 2020-11-30 19:52:19,787 INFO DistributedDataParallel( (module): SECONDNet( (vfe): MeanVFE() (backbone_3d): VoxelBackBone8x( (conv_input): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (conv1): SparseSequential( (0): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv2): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv3): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv4): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv_out): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (map_to_bev_module): HeightCompression() (pfe): None (backbone_2d): BaseBEVBackbone( (blocks): ModuleList( (0): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False) (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) (1): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) ) (deblocks): ModuleList( (0): Sequential( (0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (dense_head): AnchorHeadSingle( (cls_loss_func): SigmoidFocalClassificationLoss() (reg_loss_func): WeightedSmoothL1Loss() (dir_loss_func): WeightedCrossEntropyLoss() (conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1)) (conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1)) (conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1)) ) (point_head): None (roi_head): None ) ) 2020-11-30 19:52:19,789 INFO **Start training data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)** 2020-12-01 03:39:10,303 INFO **End training data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)**

2020-12-01 03:39:10,304 INFO **Start evaluation data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)** 2020-12-01 03:39:10,480 INFO Loading Waymo dataset 2020-12-01 03:39:13,718 INFO Total skipped info 0 2020-12-01 03:39:13,719 INFO Total samples for Waymo dataset: 39987 2020-12-01 03:39:13,720 INFO Total sampled samples for Waymo dataset: 7998 2020-12-01 03:39:13,978 INFO ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_20.pth to CPU 2020-12-01 03:39:14,051 INFO ==> Checkpoint trained from version: pcdet+0.3.0+0000000 2020-12-01 03:39:14,182 INFO ==> Done (loaded 163/163) 2020-12-01 03:39:14,184 INFO * EPOCH 20 EVALUATION *** 2020-12-01 03:40:00,176 INFO * Performance of EPOCH 20 *** 2020-12-01 03:40:00,176 INFO Generate label finished(sec_per_example: 0.0057 second). 2020-12-01 03:40:00,176 INFO recall_roi_0.3: 0.000000 2020-12-01 03:40:00,176 INFO recall_rcnn_0.3: 0.646826 2020-12-01 03:40:00,176 INFO recall_roi_0.5: 0.000000 2020-12-01 03:40:00,176 INFO recall_rcnn_0.5: 0.527019 2020-12-01 03:40:00,176 INFO recall_roi_0.7: 0.000000 2020-12-01 03:40:00,176 INFO recall_rcnn_0.7: 0.265897 2020-12-01 03:40:00,182 INFO Average predicted number of objects(7998 samples): 105.472 2020-12-01 03:44:04,417 INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.3743 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.3658 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3248 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3173 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3563 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2631 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3046 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2246 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2470 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2303 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2379 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2219

2020-12-01 03:44:04,418 INFO Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_20/val 2020-12-01 03:44:04,419 INFO ****Evaluation done.** 2020-12-01 03:44:04,425 INFO Epoch 20 has been evaluated 2020-12-01 03:44:04,427 INFO ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_25.pth to CPU 2020-12-01 03:44:04,483 INFO ==> Checkpoint trained from version: pcdet+0.3.0+0000000 2020-12-01 03:44:04,609 INFO ==> Done (loaded 163/163) 2020-12-01 03:44:04,611 INFO EPOCH 25 EVALUATION ** 2020-12-01 03:44:51,699 INFO Performance of EPOCH 25 ***** 2020-12-01 03:44:51,700 INFO Generate label finished(sec_per_example: 0.0059 second). 2020-12-01 03:44:51,701 INFO recall_roi_0.3: 0.000000 2020-12-01 03:44:51,701 INFO recall_rcnn_0.3: 0.664714 2020-12-01 03:44:51,701 INFO recall_roi_0.5: 0.000000 2020-12-01 03:44:51,701 INFO recall_rcnn_0.5: 0.543964 2020-12-01 03:44:51,701 INFO recall_roi_0.7: 0.000000 2020-12-01 03:44:51,701 INFO recall_rcnn_0.7: 0.284320 2020-12-01 03:44:51,710 INFO Average predicted number of objects(7998 samples): 100.893 2020-12-01 03:47:44,519 INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4097 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4028 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3567 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3505 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3829 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2832 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3247 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2400 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2680 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2518 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2581 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2425

2020-12-01 03:47:44,520 INFO Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_25/val 2020-12-01 03:47:44,520 INFO ****Evaluation done.** 2020-12-01 03:47:44,553 INFO Epoch 25 has been evaluated 2020-12-01 03:47:44,554 INFO ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_30.pth to CPU 2020-12-01 03:47:44,604 INFO ==> Checkpoint trained from version: pcdet+0.3.0+0000000 2020-12-01 03:47:44,737 INFO ==> Done (loaded 163/163) 2020-12-01 03:47:44,740 INFO EPOCH 30 EVALUATION ** 2020-12-01 03:48:32,127 INFO Performance of EPOCH 30 ***** 2020-12-01 03:48:32,128 INFO Generate label finished(sec_per_example: 0.0059 second). 2020-12-01 03:48:32,128 INFO recall_roi_0.3: 0.000000 2020-12-01 03:48:32,128 INFO recall_rcnn_0.3: 0.668910 2020-12-01 03:48:32,128 INFO recall_roi_0.5: 0.000000 2020-12-01 03:48:32,128 INFO recall_rcnn_0.5: 0.552218 2020-12-01 03:48:32,128 INFO recall_roi_0.7: 0.000000 2020-12-01 03:48:32,128 INFO recall_rcnn_0.7: 0.297340 2020-12-01 03:48:32,133 INFO Average predicted number of objects(7998 samples): 101.049 2020-12-01 03:51:04,877 INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4284 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4213 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3733 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3670 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4020 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3015 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3456 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2589 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2885 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2714 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2791 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2626

2020-12-01 03:51:04,878 INFO Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_30/val 2020-12-01 03:51:04,878 INFO ****Evaluation done.***** 2020-12-01 03:51:04,910 INFO Epoch 30 has been evaluated 2020-12-01 03:51:34,941 INFO **End evaluation data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)**

[sshaoshuai]

It seems that your data has some problems, the number of gt_database filter should be similar with the following:

 129 2020-11-24 15:55:00,019   INFO  Database filter by min points Vehicle: 477761 => 392931
 130 2020-11-24 15:55:00,064   INFO  Database filter by min points Pedestrian: 222675 => 175422
 131 2020-11-24 15:55:00,069   INFO  Database filter by min points Cyclist: 5344 => 4638
 132 2020-11-24 15:55:00,220   INFO  Database filter by difficulty Vehicle: 392931 => 392931
 133 2020-11-24 15:55:00,286   INFO  Database filter by difficulty Pedestrian: 175422 => 175422
 134 2020-11-24 15:55:00,288   INFO  Database filter by difficulty Cyclist: 4638 => 4638
 135 2020-11-24 15:55:00,670   INFO  Loading Waymo dataset
 136 2020-11-24 15:55:17,695   INFO  Total skipped info 0
 137 2020-11-24 15:55:17,696   INFO  Total samples for Waymo dataset: 158081
 138 2020-11-24 15:55:17,702   INFO  Total sampled samples for Waymo dataset: 31617

It may be caused by your enviroment of tensorflow and waymo_open_dataset. Please try to re-create a new enviroment with the following command (You could change the version if you find there are some problems of your current version):

# tf 2.1.0.
pip install waymo-open-dataset-tf-2-1-0==1.2.0
# tf 2.0.0
# pip install waymo-open-dataset-tf-2-0-0==1.2.0
# tf 1.15.0
# pip install waymo-open-dataset-tf-1-15-0==1.2.0

[LZDSJTU]

It seems that your data has some problems, the number of gt_database filter should be similar with the following:

 129 2020-11-24 15:55:00,019   INFO  Database filter by min points Vehicle: 477761 => 392931
 130 2020-11-24 15:55:00,064   INFO  Database filter by min points Pedestrian: 222675 => 175422
 131 2020-11-24 15:55:00,069   INFO  Database filter by min points Cyclist: 5344 => 4638
 132 2020-11-24 15:55:00,220   INFO  Database filter by difficulty Vehicle: 392931 => 392931
 133 2020-11-24 15:55:00,286   INFO  Database filter by difficulty Pedestrian: 175422 => 175422
 134 2020-11-24 15:55:00,288   INFO  Database filter by difficulty Cyclist: 4638 => 4638
 135 2020-11-24 15:55:00,670   INFO  Loading Waymo dataset
 136 2020-11-24 15:55:17,695   INFO  Total skipped info 0
 137 2020-11-24 15:55:17,696   INFO  Total samples for Waymo dataset: 158081
 138 2020-11-24 15:55:17,702   INFO  Total sampled samples for Waymo dataset: 31617

It may be caused by your enviroment of tensorflow and waymo_open_dataset. Please try to re-create a new enviroment with the following command (You could change the version if you find there are some problems of your current version):

# tf 2.1.0.
pip install waymo-open-dataset-tf-2-1-0==1.2.0
# tf 2.0.0
# pip install waymo-open-dataset-tf-2-0-0==1.2.0
# tf 1.15.0
# pip install waymo-open-dataset-tf-1-15-0==1.2.0

Yes, using tf 2.0.0 will cause wrong preprocessed results. Using tf2.1.1 seems to be right. I will re-process the waymo dataset and train the model.

[MartinHahner]

Am I getting you right, that the instructions listed here are not correct and one has to use pip install waymo-open-dataset-tf-2-1-0==1.2.0 instead of pip3 install waymo-open-dataset-tf-2-0-0==1.2.0?

[sshaoshuai]

Not exactly. Actually I tried to create a new environment and use the following for generating data:

pip3 install waymo-open-dataset-tf-2-0-0==1.2.0

and I could get the results correctly.

You could visualize some generated xxxx.npy files to see whether the point clouds are reasonable. If so, your environment should be fine.

I give a reference here, you could find the following file:

waymo_processed_data/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy

and load it with the following line:

A = np.load('0000.npy')
print(A.shape)
print(A.sum())

The results should be (171606, 6) and 609530.0.

If these numbers are matched, your data should be fine.

[MartinHahner]

@sshaoshuai Thx! My numbers match with yours and I used pip3 install waymo-open-dataset-tf-2-0-0==1.2.0 from the Getting Started page.

My waymo_processed_data is still getting copied to AWS since I had issues generating it there. Once all necessary data is there, I will try to validate your results as mentioned before.

[JunrQ]

I followed the readme and tried to reproduce the results. I trained a Part-A^2-Anchor and tested it using the tools/test.py Before the bugfix: V 1 : AP / APH | 2 : AP / APH P 1 : AP / APH | 2 : AP / APH C 1 : AP / APH | 2 : AP / APH 0.6197 / 0.6150 | 0.5721 / 0.5677 0.5428 / 0.4706 | 0.4981 / 0.4312 0.5856 / 0.5751 | 0.5846 / 0.5741

After the bugfix: 0.7198 / 0.7144 | 0.6447 / 0.6396 0.6348 / 0.5508 | 0.5454 / 0.4722 0.6562 / 0.6444 | 0.6314 / 0.6201

which is very close to the reported result 71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35

[jianrenw]

Not exactly. Actually I tried to create a new environment and use the following for generating data:

pip3 install waymo-open-dataset-tf-2-0-0==1.2.0

and I could get the results correctly.

You could visualize some generated xxxx.npy files to see whether the point clouds are reasonable. If so, your environment should be fine.

I give a reference here, you could find the following file:

waymo_processed_data/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy

and load it with the following line:

A = np.load('0000.npy')
print(A.shape)
print(A.sum())

The results should be (171606, 6) and 609530.0.

If these numbers are matched, your data should be fine.

I tried all versions of waymo-open-dataset, the A.sum()s are all different. But none of them equal to yours. Do you have any idea what might be the problem?

[jialeli1]

I followed the readme and tried to reproduce the results. I trained a Part-A^2-Anchor and tested it using the tools/test.py Before the bugfix: V 1 : AP / APH | 2 : AP / APH P 1 : AP / APH | 2 : AP / APH C 1 : AP / APH | 2 : AP / APH 0.6197 / 0.6150 | 0.5721 / 0.5677 0.5428 / 0.4706 | 0.4981 / 0.4312 0.5856 / 0.5751 | 0.5846 / 0.5741

After the bugfix: 0.7198 / 0.7144 | 0.6447 / 0.6396 0.6348 / 0.5508 | 0.5454 / 0.4722 0.6562 / 0.6444 | 0.6314 / 0.6201

which is very close to the reported result 71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35

Hi, do you mean this bugfix #384 ? if I need to re-create the val info for the waymo dataset, whether I only need to keep the line333-349 in the function _create_waymoinfos() of _waymodataset.py? Because train info is too time-consuming, I want to be sure I will not modify it.

https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/datasets/waymo/waymo_dataset.py#L309

By the way, could anyone tell me how to get the BEV and range-based metric on waymo? @MartinHahner @sshaoshuai

Thanks.

[MartinHahner]

I want to confirm (most of) the numbers given in README.md.

Numbers provided by OpenPCDet on the full evaluation set:		Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	68.03/67.44	59.57/59.04	61.14/50.33	53.00/43.56	54.66/53.31	52.67/51.37
Part-A^2-Anchor	71.82/71.29	64.33/63.82	63.15/54.96	54.24/47.11	65.23/63.92	62.61/61.35
PV-RCNN	74.06/73.38	64.99/64.38	62.66/52.68	53.80/45.14	63.32/61.71	60.72/59.18

My training runs trained and evaluated on 1/5th of the data:		Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	67.85/67.24	59.35/58.80	57.94/47.85	49.93/41.18	55.54/54.12	53.75/52.38
Part-A^2-Anchor	71.57/71.03	64.08/63.57	63.34/55.08	54.39/47.20	64.93/63.71	62.53/61.36
PV-RCNN	74.36/73.68	65.27/64.67	63.29/53.04	54.36/45.46	64.02/62.07	61.53/59.66

These are the deltas:		Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	-0.18/-0.20	-0.22/-0.24	-3.20/-2.48	-3.07/-2.38	+0.88/+0.81	+1.08/+1.01
Part-A^2-Anchor	-0.25/-0.26	-0.25/-0.25	+0.19/+0.12	+0.15/+0.09	-0.30/-0.21	-0.08/+0.01
PV-RCNN	+0.30/+0.30	+0.28/+0.31	+0.63/+0.36	+0.56/+0.32	+0.70/+0.36	+0.81/+0.48

The evaluation on the full validation set is still ongoing, but the trend goes in the same direction. Depicted are two PV-RCNN runs (Step = Epoch). Grey: trained and evaluated on 1/5th of the data. Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

[MartinHahner]

This is a comparison between an evaluation done on 1/5th of the validation set (in green) and an evaluation done on the complete validation set (in blue and pink [continued]) of a Part-A^2-Anchor training run on 1/5th of the training set.

So, in the end, there is almost no difference between the two final numbers.

[w111liang222]

I want to confirm (most of) the numbers given in README.md.

Numbers provided by OpenPCDet on the full evaluation set:

Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND 68.03/67.44 59.57/59.04 61.14/50.33 53.00/43.56 54.66/53.31 52.67/51.37 Part-A^2-Anchor 71.82/71.29 64.33/63.82 63.15/54.96 54.24/47.11 65.23/63.92 62.61/61.35 PV-RCNN 74.06/73.38 64.99/64.38 62.66/52.68 53.80/45.14 63.32/61.71 60.72/59.18 My training runs trained and evaluated on 1/5th of the data:

Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND 67.85/67.24 59.35/58.80 57.94/47.85 49.93/41.18 55.54/54.12 53.75/52.38 Part-A^2-Anchor 71.57/71.03 64.08/63.57 63.34/55.08 54.39/47.20 64.93/63.71 62.53/61.36 PV-RCNN 74.36/73.68 65.27/64.67 63.29/53.04 54.36/45.46 64.02/62.07 61.53/59.66 These are the deltas:

Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND -0.18/-0.20 -0.22/-0.24 -3.20/-2.48 -3.07/-2.38 +0.88/+0.81 +1.08/+1.01 Part-A^2-Anchor -0.25/-0.26 -0.25/-0.25 +0.19/+0.12 +0.15/+0.09 -0.30/-0.21 -0.08/+0.01 PV-RCNN +0.30/+0.30 +0.28/+0.31 +0.63/+0.36 +0.56/+0.32 +0.70/+0.36 +0.81/+0.48 The evaluation on the full validation set is still ongoing, but the trend goes in the same direction. Depicted are two PV-RCNN runs (Step = Epoch). Grey: trained and evaluated on 1/5th of the data. Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

@MartinHahner I trained the SECOND with the default configuration and test the model on the whole validation, and I got the similar results with yours. The numbers of PED L1/L2 are lower than the numbers given in README.md about 3%. Did you find out the problem? Thanks.

[MartinHahner]

If you trained SECOND once and you observe this, I think it is within the "normal" variance of different training runs. If you let's say trained SECOND five times, and every time it is by 3% lower than the reported numbers, then I don't know why.

[HenryJunW]

I want to confirm (most of) the numbers given in README.md.

Numbers provided by OpenPCDet on the full evaluation set: Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND 68.03/67.44 59.57/59.04 61.14/50.33 53.00/43.56 54.66/53.31 52.67/51.37 Part-A^2-Anchor 71.82/71.29 64.33/63.82 63.15/54.96 54.24/47.11 65.23/63.92 62.61/61.35 PV-RCNN 74.06/73.38 64.99/64.38 62.66/52.68 53.80/45.14 63.32/61.71 60.72/59.18

My training runs trained and evaluated on 1/5th of the data: Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND 67.85/67.24 59.35/58.80 57.94/47.85 49.93/41.18 55.54/54.12 53.75/52.38 Part-A^2-Anchor 71.57/71.03 64.08/63.57 63.34/55.08 54.39/47.20 64.93/63.71 62.53/61.36 PV-RCNN 74.36/73.68 65.27/64.67 63.29/53.04 54.36/45.46 64.02/62.07 61.53/59.66

These are the deltas: Vec_L1 Vec_L2 Ped_L1 Ped_L2 Cyc_L1 Cyc_L2 SECOND -0.18/-0.20 -0.22/-0.24 -3.20/-2.48 -3.07/-2.38 +0.88/+0.81 +1.08/+1.01 Part-A^2-Anchor -0.25/-0.26 -0.25/-0.25 +0.19/+0.12 +0.15/+0.09 -0.30/-0.21 -0.08/+0.01 PV-RCNN +0.30/+0.30 +0.28/+0.31 +0.63/+0.36 +0.56/+0.32 +0.70/+0.36 +0.81/+0.48

The evaluation on the full validation set is still ongoing, but the trend goes in the same direction. Depicted are two PV-RCNN runs (Step = Epoch). Grey: trained and evaluated on 1/5th of the data. Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

Thanks for sharing the reproduce results. How many GPUs are you using to reproduce the pv_rcnn results? Thanks.

[MartinHahner]

I think I used 4 or 8 GPUs. This shouldn't matter though, it would only have an impact on the training time.

[HenryJunW]

Thanks for your reply! Since I am trying to reproduce pv-rcnn results on 4 P6000 GPUs, but I am getting ~0.61 at epoch 24, which is way below what you get. I am wondering what the problem could be. Are you using the default evaluation code w/o anything modification? https://github.com/open-mmlab/OpenPCDet/blob/0642cf06d0fd84f50cc4c6c01ea28edbc72ea810/pcdet/datasets/waymo/waymo_eval.py#L239 indicates the dist_threshold to be 1000, so I am not sure whether you change it to be 30/50 to be consistent with the Waymo official metric. Thank you!

[MartinHahner]

Nope, I did not change anything.

[HenryJunW]

Thanks for your information!

[xuanzhangyang]

training pv-rcnn on waymo dataset:

/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply03:41, 2.71s/it, total_it=89722] points[:, :3] = noise_scale epochs: 90%|██████████████████████████████████████████████████████████████████████████████▎ | 27/30 [70:45:38<7:52:40, 9453.55s/it, loss=1.66, lr=0.000627]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply19:00, 2.68s/it, total_it=93601] points[:, :3] = noise_scale epochs: 93%|█████████████████████████████████████████████████████████████████████████████████▏ | 28/30 [73:22:38<5:15:11, 9455.94s/it, loss=1.56, lr=0.000274]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply20:07, 2.69s/it, total_it=97040] points[:, :3] = noise_scale epochs: 93%|█████████████████████████████████████████████████████████████████████████████████▏ | 28/30 [74:20:43<5:15:11, 9455.94s/it, loss=2.27, lr=0.000178]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply21:23, 2.66s/it, total_it=98332] points[:, :3] = noise_scale train: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 3454/3454 [2:35:10<00:00, 2.70s/it, total_it=103620] epochs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [78:20:16<00:00, 9400.56s/it, loss=2.79, lr=1e-7] 2021-03-30 17:18:14,818 INFO **End training waymo_models/pv_rcnn(default)**

epoch 30:

2021-03-30 19:22:49,337 INFO recall_roi_0.3: 0.613171 2021-03-30 19:22:49,337 INFO recall_rcnn_0.3: 0.613844 2021-03-30 19:22:49,337 INFO recall_roi_0.5: 0.550942 2021-03-30 19:22:49,337 INFO recall_rcnn_0.5: 0.562655 2021-03-30 19:22:49,337 INFO recall_roi_0.7: 0.308685 2021-03-30 19:22:49,337 INFO recall_rcnn_0.7: 0.365962

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.5073 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4977 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.4482 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.4395 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4421 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3412 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3704 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2857 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.3261 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.3033 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.3154 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2933

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[r-sy]

Hi, excuse me，waymo_processed_data_v0_5_0/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy generated by waymo-open-dataset-tf-2-5-0 in the latest version v0.5.2 is still the same as before? However, load it with the following line:

A = np.load('0000.npy')
print(A.shape)
print(A.sum())

I get the results are (183680, 6) and 586412.5. Is there something wrong with my generation?

Is this the reason why I can't fully reproduce the accuracy of PV_RCNN and other algorithms on Waymo dataset? Three or four points lower on average.

I hope you can get a reply and the problem can be solved. Thank you!

I get pv_rcnn :

2022-03-13 23:23:58,300   INFO  *************** EPOCH 30 EVALUATION *****************
2022-03-13 23:52:17,842   INFO  *************** Performance of EPOCH 30 *****************
2022-03-13 23:52:17,858   INFO  Generate label finished(sec_per_example: 0.0425 second).
2022-03-13 23:52:17,858   INFO  recall_roi_0.3: 0.804661
2022-03-13 23:52:17,858   INFO  recall_rcnn_0.3: 0.805084
2022-03-13 23:52:17,858   INFO  recall_roi_0.5: 0.741911
2022-03-13 23:52:17,858   INFO  recall_rcnn_0.5: 0.751780
2022-03-13 23:52:17,859   INFO  recall_roi_0.7: 0.486838
2022-03-13 23:52:17,859   INFO  recall_rcnn_0.7: 0.533652
2022-03-13 23:52:17,884   INFO  Average predicted number of objects(39987 samples): 88.996
2022-03-14 00:03:08,822   INFO  
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.7297 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.7227 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.6441 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.6377 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.6967 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.5787 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.6053 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.5013 
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.6462 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.6244 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.6222 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.6013

pv_rcnn_center:

2022-03-19 12:57:09,953   INFO  **********************Start evaluation waymo_models/pv_rcnn_with_centerhead_rpn(default)**********************
2022-03-19 12:57:09,955   INFO  Loading Waymo dataset
2022-03-19 12:57:22,970   INFO  Total skipped info 0
2022-03-19 12:57:22,971   INFO  Total samples for Waymo dataset: 39987
2022-03-19 12:57:22,973   INFO  ==> Loading parameters from checkpoint ./OpenPCDet/output/waymo_models/pv_rcnn_with_centerhead_rpn/default/ckpt/checkpoint_epoch_30.pth to CPU
2022-03-19 12:57:23,098   INFO  ==> Checkpoint trained from version: pcdet+0.5.2+a9c66fe
2022-03-19 12:57:23,154   INFO  ==> Done (loaded 437/437)
2022-03-19 12:57:23,158   INFO  *************** EPOCH 30 EVALUATION *****************
2022-03-19 13:28:13,964   INFO  *************** Performance of EPOCH 30 *****************
2022-03-19 13:28:14,026   INFO  Generate label finished(sec_per_example: 0.0463 second).
2022-03-19 13:28:14,026   INFO  recall_roi_0.3: 0.834954
2022-03-19 13:28:14,026   INFO  recall_rcnn_0.3: 0.835461
2022-03-19 13:28:14,027   INFO  recall_roi_0.5: 0.762491
2022-03-19 13:28:14,027   INFO  recall_rcnn_0.5: 0.772247
2022-03-19 13:28:14,027   INFO  recall_roi_0.7: 0.473409
2022-03-19 13:28:14,027   INFO  recall_rcnn_0.7: 0.532824
2022-03-19 13:28:14,058   INFO  Average predicted number of objects(39987 samples): 86.808
2022-03-19 13:43:13,291   INFO  
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.7393 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.7338 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.6538 
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.6488 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.7248 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.6550 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.6359 
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.5732 
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.6900 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.6762 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.6646 
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.6513 

[sshaoshuai]

@r-sy , Hi, the data has been updated after version 0.5 and your current data state is consistent with me. So you may check other reasons to match the performance.

[r-sy]

Ok，thank you very much！---- Replied Message ----FromShaoshuai @.>Date03/19/2022 20:22 @.> @.**@.>SubjectRe: [open-mmlab/OpenPCDet] Low accuracy when training on waymo dataset (#370) @r-sy , Hi, the data has been updated after version 0.5 and your current data state is consistent with me. So you may check other reasons to match the performance.

—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.> [ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/open-mmlab/OpenPCDet/issues/370#issuecomment-1073000369", "url": "https://github.com/open-mmlab/OpenPCDet/issues/370#issuecomment-1073000369", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[jwma0725]

@r-sy , Hi, the data has been updated after version 0.5 and your current data state is consistent with me. So you may check other reasons to match the performance. @sshaoshuai Hi, I also use the new version 0.5 to generate the waymo dataset(waymo version 1.0) with waymo-open-dataset-tf-2-5-0-1.4.1. But I do not obtain the same result.
I think the problem is the numpy version. The waymo-open-dataset-tf-2-5-0 need numpy==1.19.5. I statisfy this version, but occur error: "ImportError: numpy.core.multiarray failed to import". And this error comes from "import SharedArray" . Based on this error, I update numpy version to 1.20, this error does not occur. After I trained Voxel-RCNN and val, another error occur:"can not convert a symbolic Tensor(strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a Numpy call. which is not supported" . I return to the origninal numpy version==1.19.5 and try to delete these code about "SharedArray", the code could run but all the result are zero.(Start the waymo evaluation... Number:(pd, 1322657)VS. (gt,0)) also generates. Can you give me some advice on these problems? Thank u very much!!!

[jwma0725]

@sshaoshuai Hi, I also use the new version 0.5 to generate the waymo dataset(waymo version 1.0) with waymo-open-dataset-tf-2-5-0-1.4.1. But I do not obtain the same result.
I think the problem is the numpy version. The waymo-open-dataset-tf-2-5-0 need numpy==1.19.5. I statisfy this version, but occur error: "ImportError: numpy.core.multiarray failed to import". And this error comes from "import SharedArray" . Based on this error, I update numpy version to 1.20, this error does not occur. After I trained Voxel-RCNN and val, another error occur:"can not convert a symbolic Tensor(strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a Numpy call. which is not supported" . I return to the origninal numpy version==1.19.5 and try to delete these code about "SharedArray", the code could run but all the result are zero.(Start the waymo evaluation... Number:(pd, 1322657)VS. (gt,0)) also generates. Can you give me some advice on these problems? Thank u very much!!!

[jwma0725]

@r-sy Hi, I also use waymo-open-dataset-tf-2-5-0 and the numpy version is 1.19.5. But I do not run code nomally and generate this error:"ImportError: numpy.core.multiarray failed to import". Can u tell me your numpy version or other suggestions? Thank u very much!!!