all the AP is 0 when training on waymo dataset

[duanchengwen]

here is the data i used ├── training_0000.tar ├── training_0001.tar ├── training_0002.tar ├── validation_0000.tar ├── validation_0001.tar

the training this data on second model with the latest codes. but all the accuracy is 0

`2020-12-22 21:01:49,072 INFO **Start logging** 2020-12-22 21:01:49,072 INFO CUDA_VISIBLE_DEVICES=ALL 2020-12-22 21:01:49,072 INFO cfg_file cfgs/waymo_models/second.yaml 2020-12-22 21:01:49,072 INFO batch_size 1 2020-12-22 21:01:49,072 INFO epochs 30 2020-12-22 21:01:49,072 INFO workers 0 2020-12-22 21:01:49,072 INFO extra_tag default 2020-12-22 21:01:49,072 INFO ckpt None 2020-12-22 21:01:49,073 INFO pretrained_model None 2020-12-22 21:01:49,073 INFO launcher none 2020-12-22 21:01:49,073 INFO tcp_port 18888 2020-12-22 21:01:49,073 INFO sync_bn False 2020-12-22 21:01:49,073 INFO fix_random_seed False 2020-12-22 21:01:49,073 INFO ckpt_save_interval 1 2020-12-22 21:01:49,073 INFO local_rank 0 2020-12-22 21:01:49,073 INFO max_ckpt_save_num 30 2020-12-22 21:01:49,073 INFO merge_all_iters_to_one_epoch False 2020-12-22 21:01:49,073 INFO set_cfgs None 2020-12-22 21:01:49,073 INFO max_waiting_mins 0 2020-12-22 21:01:49,073 INFO start_epoch 0 2020-12-22 21:01:49,073 INFO save_to_file False 2020-12-22 21:01:49,073 INFO cfg.ROOT_DIR: /home/zombie/zombie/deepleaning/lidar/OpenPCDet 2020-12-22 21:01:49,073 INFO cfg.LOCAL_RANK: 0 2020-12-22 21:01:49,073 INFO cfg.CLASS_NAMES: ['Vehicle', 'Pedestrian', 'Cyclist'] 2020-12-22 21:01:49,073 INFO
cfg.DATA_CONFIG = edict() 2020-12-22 21:01:49,073 INFO cfg.DATA_CONFIG.DATASET: WaymoDataset 2020-12-22 21:01:49,073 INFO cfg.DATA_CONFIG.DATA_PATH: /media/zombie/DATA/waymo_dataset 2020-12-22 21:01:49,073 INFO cfg.DATA_CONFIG.PROCESSED_DATA_TAG: waymo_processed_data 2020-12-22 21:01:49,073 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4] 2020-12-22 21:01:49,073 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict() 2020-12-22 21:01:49,073 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val 2020-12-22 21:01:49,074 INFO
cfg.DATA_CONFIG.SAMPLED_INTERVAL = edict() 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.SAMPLED_INTERVAL.train: 5 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.SAMPLED_INTERVAL.test: 5 2020-12-22 21:01:49,074 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict() 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder'] 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['pcdet_waymo_dbinfos_train_sampled_10.pkl'], 'PREPARE': {'filter_by_min_points': ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Vehicle:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 5, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}] 2020-12-22 21:01:49,074 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict() 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity', 'elongation'] 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation'] 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 80000, 'test': 90000}}] 2020-12-22 21:01:49,074 INFO cfg.DATA_CONFIG._BASECONFIG: /home/zombie/zombie/deepleaning/lidar/OpenPCDet/tools/cfgs/dataset_configs/waymo_dataset.yaml 2020-12-22 21:01:49,074 INFO
cfg.MODEL = edict() 2020-12-22 21:01:49,074 INFO cfg.MODEL.NAME: SECONDNet 2020-12-22 21:01:49,074 INFO
cfg.MODEL.VFE = edict() 2020-12-22 21:01:49,074 INFO cfg.MODEL.VFE.NAME: MeanVFE 2020-12-22 21:01:49,074 INFO
cfg.MODEL.BACKBONE_3D = edict() 2020-12-22 21:01:49,074 INFO cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x 2020-12-22 21:01:49,074 INFO
cfg.MODEL.MAP_TO_BEV = edict() 2020-12-22 21:01:49,074 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression 2020-12-22 21:01:49,074 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256 2020-12-22 21:01:49,074 INFO
cfg.MODEL.BACKBONE_2D = edict() 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5] 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2] 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256] 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2] 2020-12-22 21:01:49,075 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256] 2020-12-22 21:01:49,075 INFO
cfg.MODEL.DENSE_HEAD = edict() 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Vehicle', 'anchor_sizes': [[4.7, 2.1, 1.7]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.55, 'unmatched_threshold': 0.4}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.91, 0.86, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.78, 0.84, 1.78]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}] 2020-12-22 21:01:49,075 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict() 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder 2020-12-22 21:01:49,075 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict() 2020-12-22 21:01:49,075 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2020-12-22 21:01:49,075 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0 2020-12-22 21:01:49,076 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0 2020-12-22 21:01:49,076 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2 2020-12-22 21:01:49,076 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 2020-12-22 21:01:49,076 INFO
cfg.MODEL.POST_PROCESSING = edict() 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7] 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: waymo 2020-12-22 21:01:49,076 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict() 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096 2020-12-22 21:01:49,076 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500 2020-12-22 21:01:49,076 INFO
cfg.OPTIMIZATION = edict() 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 30 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.LR: 0.003 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85] 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.PCT_START: 0.4 2020-12-22 21:01:49,076 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45] 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.LR_WARMUP: False 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1 2020-12-22 21:01:49,077 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10 2020-12-22 21:01:49,077 INFO cfg.TAG: second 2020-12-22 21:01:49,077 INFO cfg.EXP_GROUP_PATH: waymo_models 2020-12-22 21:01:49,287 INFO Database filter by min points Vehicle: 45122 => 37953 2020-12-22 21:01:49,288 INFO Database filter by min points Pedestrian: 7205 => 5919 2020-12-22 21:01:49,288 INFO Database filter by min points Cyclist: 312 => 287 2020-12-22 21:01:49,296 INFO Database filter by difficulty Vehicle: 37953 => 37953 2020-12-22 21:01:49,298 INFO Database filter by difficulty Pedestrian: 5919 => 5919 2020-12-22 21:01:49,298 INFO Database filter by difficulty Cyclist: 287 => 287 2020-12-22 21:01:49,359 INFO Loading Waymo dataset 2020-12-22 21:01:49,859 INFO Total skipped info 0 2020-12-22 21:01:49,859 INFO Total samples for Waymo dataset: 14486 2020-12-22 21:01:49,859 INFO Total sampled samples for Waymo dataset: 2898 2020-12-22 21:01:52,286 INFO ==> Loading parameters from checkpoint /home/zombie/zombie/deepleaning/lidar/OpenPCDet/output/waymo_models/second/default/ckpt/checkpoint_epoch_5.pth to CPU 2020-12-22 21:01:52,341 INFO ==> Loading optimizer parameters from checkpoint /home/zombie/zombie/deepleaning/lidar/OpenPCDet/output/waymo_models/second/default/ckpt/checkpoint_epoch_5.pth to CPU ==> Checkpoint trained from version: pcdet+0.3.0+0e84f82 2020-12-22 21:01:52,379 INFO ==> Done 2020-12-22 21:01:52,382 INFO SECONDNet( (vfe): MeanVFE() (backbone_3d): VoxelBackBone8x( (conv_input): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (conv1): SparseSequential( (0): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv2): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv3): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv4): SparseSequential( (0): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d() (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv_out): SparseSequential( (0): SparseConv3d() (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (map_to_bev_module): HeightCompression() (pfe): None (backbone_2d): BaseBEVBackbone( (blocks): ModuleList( (0): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False) (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) (1): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) ) (deblocks): ModuleList( (0): Sequential( (0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (dense_head): AnchorHeadSingle( (cls_loss_func): SigmoidFocalClassificationLoss() (reg_loss_func): WeightedSmoothL1Loss() (dir_loss_func): WeightedCrossEntropyLoss() (conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1)) (conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1)) (conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1)) ) (point_head): None (roi_head): None ) 2020-12-22 21:01:52,383 INFO **Start training waymo_models/second(default)** epochs: 0%| | 0/25 [00:00<?, ?it/s/home/zombie/zombie/deepleaning/lidar/OpenPCDet/pcdet/models/dense_heads/target_assigner/axis_aligned_target_assigner.py:154: UserWarning: This overload of nonzero is deprecated: | 0/2898 [00:00<?, ?it/s] nonzero() Consider using one of the following signatures instead: nonzero(*, bool as_tuple) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.) anchors_with_max_overlap = (anchor_by_gt_overlap == gt_to_anchor_max).nonzero()[:, 0] epochs: 4%|█████▎ | 1/25 [1:04:18<13:15:51, 1989.64s/it, loss=1.22, lr=0.002]epochs: 4%|█████▎ | 1/25 [1:04:18<13:15:51, 1989.64s/it, loss=0.87, lr=0.002]train: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2898/2898 [31:03<00:00, 1.55it/s, total_it=86940] epochs: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [13:05:15<00:00, 1884.61s/it, loss=1.46, lr=3e-8] 2020-12-23 10:07:07,709 INFO **End training waymo_models/second(default)**

file == /home/zombie/zombie/deepleaning/lidar/OpenPCDet/output/waymo_models/second/default/ckpt/checkpoint_epoch_30.pth 2020-12-23 10:36:51,523 INFO ==> Loading parameters from checkpoint /home/zombie/zombie/deepleaning/lidar/OpenPCDet/output/waymo_models/second/default/ckpt/checkpoint_epoch_30.pth to GPU 2020-12-23 10:36:51,608 INFO ==> Checkpoint trained from version: pcdet+0.3.0+0e84f82 2020-12-23 10:36:51,742 INFO ==> Done (loaded 163/163) 2020-12-23 10:36:51,746 INFO * EPOCH 30 EVALUATION *** eval: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1983/1983 [02:47<00:00, 11.87it/s, recall_0.3=(0, 54150) / 74830] 2020-12-23 10:39:38,759 INFO * Performance of EPOCH 30 *** 2020-12-23 10:39:38,759 INFO Generate label finished(sec_per_example: 0.0842 second). 2020-12-23 10:39:38,759 INFO recall_roi_0.3: 0.000000 2020-12-23 10:39:38,759 INFO recall_rcnn_0.3: 0.723640 2020-12-23 10:39:38,759 INFO recall_roi_0.5: 0.000000 2020-12-23 10:39:38,760 INFO recall_rcnn_0.5: 0.640532 2020-12-23 10:39:38,760 INFO recall_roi_0.7: 0.000000 2020-12-23 10:39:38,760 INFO recall_rcnn_0.7: 0.400909 2020-12-23 10:39:38,760 INFO Average predicted number of objects(1983 samples): 58.056 Start the waymo evaluation... Number: (pd, 115125) VS. (gt, 0) Level 1: 0, Level2: 0) 2020-12-23 10:39:39.267625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-23 10:39:39.267658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]
2020-12-23 10:39:39.318131: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:39.318164: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:39.344263: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:39.460312: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:39.518959: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:39.518993: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:39.541163: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:39.655748: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:39.715529: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:39.715561: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:39.738099: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:39.852261: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:39.909590: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:39.909636: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:39.930640: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:40.057543: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:40.121250: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:40.121286: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:40.146768: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:40.264894: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:40.322628: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:40.322659: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:40.345940: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:40.463075: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:40.523954: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:40.523996: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:40.547976: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:40.661599: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:40.721689: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:40.721749: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:40.744914: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:40.856738: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:40.914816: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:40.914849: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:40.937291: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:41.055676: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:41.116400: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:41.116433: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:41.138756: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:41.256524: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:41.314279: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:41.314317: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:41.336739: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:41.460779: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:41.518792: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:41.518823: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:41.541227: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:41.656426: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:41.717149: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:41.717182: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:41.739818: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:41.853856: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:41.914684: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:41.914716: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:41.937205: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:42.052602: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:42.117211: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:42.117236: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:42.141252: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:42.262865: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:42.324895: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:157] Computing detection metrics for 115125 predicted boxes. 2020-12-23 10:39:42.324978: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:159] Parsing prediction [115125,7][115125] 2020-12-23 10:39:42.350499: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:168] Parsing ground truth [0,7][0] 2020-12-23 10:39:42.472671: I waymo_open_dataset/metrics/ops/detection_metrics_ops.cc:214] Done with computing detection metrics. 2020-12-23 10:39:42,531 INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.0000 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.0000

2020-12-23 10:39:42,531 INFO Result is save to /home/zombie/zombie/deepleaning/lidar/OpenPCDet/output/waymo_models/second/default/eval/eval_with_train/epoch_30/val 2020-12-23 10:39:42,531 INFO ****Evaluation done.***** 2020-12-23 10:39:42,538 INFO Epoch 30 has been evaluated 2020-12-23 10:40:12,565 INFO **End evaluation waymo_models/second(default)**o_models/second/default/ckpt `

[JingyaLiu]

I have the same issue. Did you solve it?

[duanchengwen]

I have the same issue. Did you solve it?

have not

[xuxuxuchen]

I have the same issue, too. Did you solve it?

[YongYongLi123]

I have the same issue, too. Did you solve it?

[Eaphan]

I have the same issue. Did you solve it?

[jihanyang]

Notice that the number of gt is 0.

2020-12-23 10:39:38,760 INFO Average predicted number of objects(1983 samples): 58.056
Start the waymo evaluation...
Number: (pd, 115125) VS. (gt, 0)

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[carrolltm]

I have the same issue. Did you solve it?

have not

I have the same issue. Did you solve it?

[fx110127]

I have the same issue. Did you solve it?

[Eaphan]

The error happened to me a long time ago. It seems that you should download the Waymo v1.2 dataset instead of v1.0 dataset. I am not 100% sure.

[carrolltm]

i try to download the Waymo v1.2 dataset instead of v1.0 dataset. And it worked.