search

CVMI-Lab / SparseKD

(NeurlPS 2022) Towards Efficient 3D Object Detection with Knowledge Distillation
Apache License 2.0
110 stars 10 forks source link

[train cp-voxel-s_sparsekd.yaml]numpy.core._exceptions.MemoryError: Unable to allocate 458. GiB for an array with shape (247985, 247985) #5

Closed HuangCongQing closed 1 year ago

HuangCongQing commented 1 year ago

python train.py --cfg_file=cfgs/waymo_models/cp-voxel/cp-voxel-s_sparsekd.yaml --batch_size=1 --epochs=20

image

/home/chongqinghuang/anaconda3/envs/pcdet/bin/python train.py --cfg_file=cfgs/waymo_models/cp-voxel/cp-voxel-s_sparsekd.yaml --batch_size=1 --epochs=20
2023-01-09 18:02:40,244   INFO  **********************Start logging**********************
2023-01-09 18:02:40,244   INFO  CUDA_VISIBLE_DEVICES=ALL
2023-01-09 18:02:40,244   INFO  cfg_file         cfgs/waymo_models/cp-voxel/cp-voxel-s_sparsekd.yaml
2023-01-09 18:02:40,244   INFO  batch_size       1
2023-01-09 18:02:40,244   INFO  epochs           20
2023-01-09 18:02:40,244   INFO  workers          4
2023-01-09 18:02:40,244   INFO  extra_tag        default
2023-01-09 18:02:40,244   INFO  ckpt             None
2023-01-09 18:02:40,244   INFO  pretrained_model /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth
2023-01-09 18:02:40,244   INFO  launcher         none
2023-01-09 18:02:40,244   INFO  tcp_port         18888
2023-01-09 18:02:40,244   INFO  sync_bn          False
2023-01-09 18:02:40,244   INFO  fix_random_seed  False
2023-01-09 18:02:40,244   INFO  ckpt_save_interval 1
2023-01-09 18:02:40,245   INFO  local_rank       0
2023-01-09 18:02:40,245   INFO  max_ckpt_save_num 30
2023-01-09 18:02:40,245   INFO  merge_all_iters_to_one_epoch False
2023-01-09 18:02:40,245   INFO  set_cfgs         None
2023-01-09 18:02:40,245   INFO  max_waiting_mins 0
2023-01-09 18:02:40,245   INFO  start_epoch      0
2023-01-09 18:02:40,245   INFO  save_to_file     False
2023-01-09 18:02:40,245   INFO  teacher_ckpt     /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth
2023-01-09 18:02:40,245   INFO  cfg.ROOT_DIR: /home/chongqinghuang/code/light_weight/SparseKD
2023-01-09 18:02:40,245   INFO  cfg.LOCAL_RANK: 0
2023-01-09 18:02:40,245   INFO  cfg.CLASS_NAMES: ['Vehicle', 'Pedestrian', 'Cyclist']
2023-01-09 18:02:40,245   INFO  cfg.TEACHER_CKPT: /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth
2023-01-09 18:02:40,245   INFO  cfg.PRETRAINED_MODEL: /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth
2023-01-09 18:02:40,245   INFO  
cfg.DATA_CONFIG = edict()
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATASET: WaymoDataset
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_PATH: ../data/waymo
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.PROCESSED_DATA_TAG: waymo_processed_data_v0_5_0
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
2023-01-09 18:02:40,245   INFO  
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_SPLIT.train: train
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_SPLIT.test: val
2023-01-09 18:02:40,245   INFO  
cfg.DATA_CONFIG.SAMPLED_INTERVAL = edict()
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.SAMPLED_INTERVAL.train: 5
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.SAMPLED_INTERVAL.test: 5
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.FILTER_EMPTY_BOXES_FOR_TRAIN: True
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DISABLE_NLZ_FLAG_ON_POINTS: True
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.USE_SHARED_MEMORY: False
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.SHARED_MEMORY_FILE_LIMIT: 35000
2023-01-09 18:02:40,245   INFO  
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl'], 'USE_SHARED_MEMORY': True, 'DB_DATA_PATH': ['waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy'], 'PREPARE': {'filter_by_min_points': ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Vehicle:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 5, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2023-01-09 18:02:40,245   INFO  
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity', 'elongation']
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation']
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 150000, 'test': 150000}}]
2023-01-09 18:02:40,245   INFO  cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/waymo_dataset.yaml
2023-01-09 18:02:40,245   INFO  
cfg.MODEL = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.NAME: CenterPoint
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.VFE = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.VFE.NAME: MeanVFE
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.BACKBONE_3D = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_3D.NAME: VoxelResBackBone8x
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_3D.ACT_FN: ReLU
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_3D.NUM_FILTERS: [16, 16, 32, 64, 128, 128]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_3D.LAYER_NUMS: [1, 2, 3, 3, 3, 1]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_3D.WIDTH: 1.0
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.MAP_TO_BEV = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2023-01-09 18:02:40,245   INFO  cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.BACKBONE_2D = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.ACT_FN: ReLU
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.NORM_TYPE: BatchNorm2d
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.WIDTH: 0.5
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.NAME: CenterHead
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.ACT_FN: ReLU
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.NORM_TYPE: BatchNorm2d
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.CLASS_NAMES_EACH_HEAD: [['Vehicle', 'Pedestrian', 'Cyclist']]
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SHARED_CONV_CHANNEL: 32
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.USE_BIAS_BEFORE_NORM: True
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.NUM_HM_CONV: 2
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_ORDER: ['center', 'center_z', 'dim', 'rot']
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT = edict()
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.out_channels: 2
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.num_conv: 2
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.out_channels: 1
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.num_conv: 2
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.out_channels: 3
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.num_conv: 2
2023-01-09 18:02:40,245   INFO  
cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot = edict()
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.out_channels: 2
2023-01-09 18:02:40,245   INFO  cfg.MODEL.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.num_conv: 2
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.FEATURE_MAP_STRIDE: 8
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NUM_MAX_OBJS: 500
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.GAUSSIAN_OVERLAP: 0.1
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MIN_RADIUS: 2
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.POST_PROCESSING = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.SCORE_THRESH: 0.1
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.POST_CENTER_LIMIT_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.MAX_OBJ_PER_SAMPLE: 500
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.7
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.LOGIT_KD = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ENABLED: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.MODE: raw_pred
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN.MODE: interpolate
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN.target: teacher
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN.mode: bilinear
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN.align_corners: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LOGIT_KD.ALIGN.align_channel: False
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD.ENABLED: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD.SCORE_TYPE: cls
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD.USE_GT: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD.GT_FIRST: False
2023-01-09 18:02:40,246   INFO  cfg.MODEL.DENSE_HEAD.LABEL_ASSIGN_KD.SCORE_THRESH: [0.6, 0.6, 0.6]
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.POST_PROCESSING = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2023-01-09 18:02:40,246   INFO  cfg.MODEL.POST_PROCESSING.EVAL_METRIC: waymo
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.POST_PROCESSING.EVAL_CLASSES = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.POST_PROCESSING.EVAL_CLASSES.LEVEL_2/AP: ['OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP', 'OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP', 'OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP']
2023-01-09 18:02:40,246   INFO  cfg.MODEL.POST_PROCESSING.EVAL_CLASSES.LEVEL_2/APH: ['OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH', 'OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH', 'OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH']
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD: True
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.KD_LOSS = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.ENABLED: True
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.KD_LOSS.HM_LOSS = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.type: MSELoss
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.weight: 10.0
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.thresh: 0.0
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.fg_mask: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.soft_mask: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.HM_LOSS.rank: -1
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.KD_LOSS.REG_LOSS = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.REG_LOSS.type: RegLossCenterNet
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.REG_LOSS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.REG_LOSS.weight: 0.2
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.KD_LOSS.FEATURE_LOSS = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.mode: rois
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.type: MSELoss
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.weight: 0.1
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.fg_mask: False
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.score_mask: False
2023-01-09 18:02:40,246   INFO  cfg.MODEL.KD_LOSS.FEATURE_LOSS.score_thresh: 0.3
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.LOGIT_KD = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ENABLED: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.MODE: raw_pred
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.LOGIT_KD.ALIGN = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ALIGN.MODE: interpolate
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ALIGN.target: teacher
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ALIGN.mode: bilinear
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ALIGN.align_corners: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LOGIT_KD.ALIGN.align_channel: False
2023-01-09 18:02:40,246   INFO  
cfg.MODEL.LABEL_ASSIGN_KD = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LABEL_ASSIGN_KD.ENABLED: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LABEL_ASSIGN_KD.SCORE_TYPE: cls
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LABEL_ASSIGN_KD.USE_GT: True
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LABEL_ASSIGN_KD.GT_FIRST: False
2023-01-09 18:02:40,246   INFO  cfg.MODEL.LABEL_ASSIGN_KD.SCORE_THRESH: [0.6, 0.6, 0.6]
2023-01-09 18:02:40,246   INFO  
cfg.MODEL_TEACHER = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.NAME: CenterPoint
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.IS_TEACHER: True
2023-01-09 18:02:40,246   INFO  
cfg.MODEL_TEACHER.VFE = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.VFE.NAME: MeanVFE
2023-01-09 18:02:40,246   INFO  
cfg.MODEL_TEACHER.BACKBONE_3D = edict()
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.BACKBONE_3D.NAME: VoxelResBackBone8x
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.BACKBONE_3D.ACT_FN: ReLU
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.BACKBONE_3D.NUM_FILTERS: [16, 16, 32, 64, 128, 128]
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.BACKBONE_3D.LAYER_NUMS: [1, 2, 3, 3, 3, 1]
2023-01-09 18:02:40,246   INFO  cfg.MODEL_TEACHER.BACKBONE_3D.WIDTH: 1.0
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.MAP_TO_BEV = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.MAP_TO_BEV.NAME: HeightCompression
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.BACKBONE_2D = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.NAME: BaseBEVBackbone
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.ACT_FN: ReLU
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.NORM_TYPE: BatchNorm2d
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.WIDTH: 1.0
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.LAYER_NUMS: [5, 5]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.NUM_FILTERS: [128, 256]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.NAME: CenterHead
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.CLASS_AGNOSTIC: False
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.ACT_FN: ReLU
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.NORM_TYPE: BatchNorm2d
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.CLASS_NAMES_EACH_HEAD: [['Vehicle', 'Pedestrian', 'Cyclist']]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SHARED_CONV_CHANNEL: 64
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.USE_BIAS_BEFORE_NORM: True
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.NUM_HM_CONV: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_ORDER: ['center', 'center_z', 'dim', 'rot']
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT = edict()
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.out_channels: 2
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center.num_conv: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.out_channels: 1
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.center_z.num_conv: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.out_channels: 3
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.dim.num_conv: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.out_channels: 2
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.SEPARATE_HEAD_CFG.HEAD_DICT.rot.num_conv: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.FEATURE_MAP_STRIDE: 8
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NUM_MAX_OBJS: 500
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.GAUSSIAN_OVERLAP: 0.1
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MIN_RADIUS: 2
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.LOSS_CONFIG = edict()
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.SCORE_THRESH: 0.1
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.POST_CENTER_LIMIT_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.MAX_OBJ_PER_SAMPLE: 500
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.7
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.DENSE_HEAD.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.POST_PROCESSING = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.POST_PROCESSING.EVAL_METRIC: waymo
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.POST_PROCESSING.EVAL_CLASSES = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.POST_PROCESSING.EVAL_CLASSES.LEVEL_2/AP: ['OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP', 'OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP', 'OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP']
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.POST_PROCESSING.EVAL_CLASSES.LEVEL_2/APH: ['OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH', 'OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH', 'OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH']
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.KD: True
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.LOGIT_KD = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ENABLED: True
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.MODE: raw_pred
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.LOGIT_KD.ALIGN = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ALIGN.MODE: interpolate
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ALIGN.target: teacher
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ALIGN.mode: bilinear
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ALIGN.align_corners: True
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LOGIT_KD.ALIGN.align_channel: False
2023-01-09 18:02:40,247   INFO  
cfg.MODEL_TEACHER.LABEL_ASSIGN_KD = edict()
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LABEL_ASSIGN_KD.ENABLED: True
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LABEL_ASSIGN_KD.SCORE_TYPE: cls
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LABEL_ASSIGN_KD.USE_GT: True
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LABEL_ASSIGN_KD.GT_FIRST: False
2023-01-09 18:02:40,247   INFO  cfg.MODEL_TEACHER.LABEL_ASSIGN_KD.SCORE_THRESH: [0.6, 0.6, 0.6]
2023-01-09 18:02:40,247   INFO  
cfg.OPTIMIZATION = edict()
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.NUM_EPOCHS: 30
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.LR: 0.003
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.MOMENTUM: 0.9
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.PCT_START: 0.4
2023-01-09 18:02:40,247   INFO  cfg.OPTIMIZATION.DIV_FACTOR: 10
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.LR_DECAY: 0.1
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.LR_CLIP: 1e-07
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.LR_WARMUP: False
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2023-01-09 18:02:40,248   INFO  
cfg.OPTIMIZATION.REMAP_PRETRAIN = edict()
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.REMAP_PRETRAIN.ENABLED: True
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.REMAP_PRETRAIN.WAY: FNAV2
2023-01-09 18:02:40,248   INFO  
cfg.OPTIMIZATION.REMAP_PRETRAIN.BN_SCALE = edict()
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.REMAP_PRETRAIN.BN_SCALE.ABS: True
2023-01-09 18:02:40,248   INFO  
cfg.OPTIMIZATION.REMAP_PRETRAIN.OFA = edict()
2023-01-09 18:02:40,248   INFO  cfg.OPTIMIZATION.REMAP_PRETRAIN.OFA.l1_norm: max
2023-01-09 18:02:40,248   INFO  
cfg.KD = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.ENABLED: True
2023-01-09 18:02:40,248   INFO  cfg.KD.TEACHER_MODE: train
2023-01-09 18:02:40,248   INFO  cfg.KD.DIFF_VOXEL: False
2023-01-09 18:02:40,248   INFO  
cfg.KD.MASK = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.MASK.SCORE_MASK: False
2023-01-09 18:02:40,248   INFO  cfg.KD.MASK.FG_MASK: False
2023-01-09 18:02:40,248   INFO  cfg.KD.MASK.BOX_MASK: False
2023-01-09 18:02:40,248   INFO  
cfg.KD.LOGIT_KD = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ENABLED: True
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.MODE: raw_pred
2023-01-09 18:02:40,248   INFO  
cfg.KD.LOGIT_KD.ALIGN = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ALIGN.MODE: interpolate
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ALIGN.target: teacher
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ALIGN.mode: bilinear
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ALIGN.align_corners: True
2023-01-09 18:02:40,248   INFO  cfg.KD.LOGIT_KD.ALIGN.align_channel: False
2023-01-09 18:02:40,248   INFO  
cfg.KD.FEATURE_KD = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ENABLED: False
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.FEATURE_NAME: spatial_features_2d
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.FEATURE_NAME_TEA: spatial_features_2d
2023-01-09 18:02:40,248   INFO  
cfg.KD.FEATURE_KD.ALIGN = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.ENABLED: False
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.MODE: interpolate
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.target: teacher
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.mode: bilinear
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.align_corners: True
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.align_channel: False
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.num_filters: [192, 384]
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.use_norm: True
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.use_act: False
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.kernel_size: 3
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ALIGN.groups: 1
2023-01-09 18:02:40,248   INFO  
cfg.KD.FEATURE_KD.ROI_POOL = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ROI_POOL.ENABLED: True
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ROI_POOL.GRID_SIZE: 7
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ROI_POOL.DOWNSAMPLE_RATIO: 1
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ROI_POOL.ROI: gt
2023-01-09 18:02:40,248   INFO  cfg.KD.FEATURE_KD.ROI_POOL.THRESH: 0.0
2023-01-09 18:02:40,248   INFO  
cfg.KD.LABEL_ASSIGN_KD = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.LABEL_ASSIGN_KD.ENABLED: True
2023-01-09 18:02:40,248   INFO  cfg.KD.LABEL_ASSIGN_KD.SCORE_TYPE: cls
2023-01-09 18:02:40,248   INFO  cfg.KD.LABEL_ASSIGN_KD.USE_GT: True
2023-01-09 18:02:40,248   INFO  cfg.KD.LABEL_ASSIGN_KD.GT_FIRST: False
2023-01-09 18:02:40,248   INFO  cfg.KD.LABEL_ASSIGN_KD.SCORE_THRESH: [0.6, 0.6, 0.6]
2023-01-09 18:02:40,248   INFO  
cfg.KD.NMS_CONFIG = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD.NMS_CONFIG.ENABLED: False
2023-01-09 18:02:40,248   INFO  cfg.KD.NMS_CONFIG.NMS_TYPE: nms_gpu
2023-01-09 18:02:40,248   INFO  cfg.KD.NMS_CONFIG.NMS_THRESH: 0.7
2023-01-09 18:02:40,248   INFO  cfg.KD.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2023-01-09 18:02:40,248   INFO  cfg.KD.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2023-01-09 18:02:40,248   INFO  
cfg.KD_LOSS = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.ENABLED: True
2023-01-09 18:02:40,248   INFO  
cfg.KD_LOSS.HM_LOSS = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.type: MSELoss
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.weight: 10.0
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.thresh: 0.0
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.fg_mask: True
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.soft_mask: True
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.HM_LOSS.rank: -1
2023-01-09 18:02:40,248   INFO  
cfg.KD_LOSS.REG_LOSS = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.REG_LOSS.type: RegLossCenterNet
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.REG_LOSS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.REG_LOSS.weight: 0.2
2023-01-09 18:02:40,248   INFO  
cfg.KD_LOSS.FEATURE_LOSS = edict()
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.mode: rois
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.type: MSELoss
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.weight: 0.1
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.fg_mask: False
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.score_mask: False
2023-01-09 18:02:40,248   INFO  cfg.KD_LOSS.FEATURE_LOSS.score_thresh: 0.3
2023-01-09 18:02:40,248   INFO  cfg.TAG: cp-voxel-s_sparsekd
2023-01-09 18:02:40,248   INFO  cfg.EXP_GROUP_PATH: waymo_models/cp-voxel
2023-01-09 18:02:40,295   INFO  Database filter by min points Vehicle: 4430 => 3909
2023-01-09 18:02:40,296   INFO  Database filter by min points Pedestrian: 3967 => 3319
2023-01-09 18:02:40,296   INFO  Database filter by min points Cyclist: 153 => 139
2023-01-09 18:02:40,296   INFO  Database filter by difficulty Vehicle: 3909 => 3909
2023-01-09 18:02:40,297   INFO  Database filter by difficulty Pedestrian: 3319 => 3319
2023-01-09 18:02:40,297   INFO  Database filter by difficulty Cyclist: 139 => 139
2023-01-09 18:02:40,297   INFO  Loading GT database to shared memory
2023-01-09 18:02:40,349   INFO  GT database has been saved to shared memory
2023-01-09 18:02:40,350   INFO  Loading Waymo dataset
2023-01-09 18:02:40,362   INFO  Total skipped info 0
2023-01-09 18:02:40,362   INFO  Total samples for Waymo dataset: 992
2023-01-09 18:02:40,362   INFO  Total sampled samples for Waymo dataset: 199
2023-01-09 18:02:42,151   INFO  Loading teacher parameters >>>>>>
2023-01-09 18:02:42,151   INFO  ==> Loading parameters from checkpoint /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth to GPU
2023-01-09 18:02:42,168   INFO  ==> Checkpoint trained from version: pcdet+0.5.2+e348415+pycb193af
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.1.weight: torch.Size([128, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.2.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.2.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.2.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.2.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.4.weight: torch.Size([128, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.5.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.5.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.5.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.5.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.7.weight: torch.Size([128, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.8.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.8.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.8.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.8.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.10.weight: torch.Size([128, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.11.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.11.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.11.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.11.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.13.weight: torch.Size([128, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.14.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.14.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.14.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.14.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.16.weight: torch.Size([128, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.17.weight: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.17.bias: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.17.running_mean: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.0.17.running_var: torch.Size([128])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.1.weight: torch.Size([256, 128, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.2.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.2.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.2.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.2.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.4.weight: torch.Size([256, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.5.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.5.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.5.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.5.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.7.weight: torch.Size([256, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.8.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.8.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.8.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.8.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.10.weight: torch.Size([256, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.11.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.11.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.11.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.11.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.13.weight: torch.Size([256, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.14.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.14.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.14.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.14.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.16.weight: torch.Size([256, 256, 3, 3])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.17.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.17.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.17.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.blocks.1.17.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.0.0.weight: torch.Size([128, 256, 1, 1])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.0.1.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.0.1.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.0.1.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.0.1.running_var: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.1.0.weight: torch.Size([256, 256, 2, 2])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.1.1.weight: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.1.1.bias: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.1.1.running_mean: torch.Size([256])
2023-01-09 18:02:42,173   INFO  Not updated weight backbone_2d.deblocks.1.1.running_var: torch.Size([256])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.0.weight: torch.Size([64, 512, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.shared_conv.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.0.weight: torch.Size([64, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.0.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center.1.weight: torch.Size([2, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.0.weight: torch.Size([64, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.0.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.center_z.1.weight: torch.Size([1, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.0.weight: torch.Size([64, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.0.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.dim.1.weight: torch.Size([3, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.0.weight: torch.Size([64, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.0.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.rot.1.weight: torch.Size([2, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.0.weight: torch.Size([64, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.0.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.1.weight: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.1.bias: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.1.running_mean: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.0.1.running_var: torch.Size([64])
2023-01-09 18:02:42,174   INFO  Not updated weight dense_head.heads_list.0.hm.1.weight: torch.Size([3, 64, 3, 3])
2023-01-09 18:02:42,174   INFO  ==> Done (loaded 168/279)
2023-01-09 18:02:42,179   INFO  Loading pretrained parameters >>>>>>
2023-01-09 18:02:42,179   INFO  ==> Loading parameters from checkpoint /home/chongqinghuang/code/light_weight/SparseKD/output/waymo_models/cp-voxel/cp-voxel-s/default/ckpt/checkpoint_epoch_4.pth to GPU
2023-01-09 18:02:42,195   INFO  ==> Checkpoint trained from version: pcdet+0.5.2+e348415+pycb193af
2023-01-09 18:02:42,195   INFO  ==> Remap pretrained model parameters with: fnav2
2023-01-09 18:02:42,203   INFO  ==> Done (loaded 279/279)
2023-01-09 18:02:42,204   INFO  CenterPoint(
  (vfe): MeanVFE()
  (backbone_3d): VoxelResBackBone8x(
    (conv_input): SparseSequential(
      (0): SubMConv3d(5, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm)
      (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (conv1): SparseSequential(
      (0): SparseBasicBlock(
        (conv1): SubMConv3d(16, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(16, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
      (1): SparseBasicBlock(
        (conv1): SubMConv3d(16, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(16, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
    )
    (conv2): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d(16, 32, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm)
        (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseBasicBlock(
        (conv1): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
      (2): SparseBasicBlock(
        (conv1): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
    )
    (conv3): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d(32, 64, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm)
        (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseBasicBlock(
        (conv1): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
      (2): SparseBasicBlock(
        (conv1): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
    )
    (conv4): SparseSequential(
      (0): SparseSequential(
        (0): SparseConv3d(64, 128, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[0, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm)
        (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): SparseBasicBlock(
        (conv1): SubMConv3d(128, 128, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(128, 128, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
      (2): SparseBasicBlock(
        (conv1): SubMConv3d(128, 128, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (relu): ReLU()
        (conv2): SubMConv3d(128, 128, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], algo=ConvAlgo.MaskImplicitGemm)
        (bn2): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      )
    )
    (conv_out): SparseSequential(
      (0): SparseConv3d(128, 128, kernel_size=[3, 1, 1], stride=[2, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm)
      (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
    )
    (final_act): ReLU()
  )
  (map_to_bev_module): HeightCompression()
  (pillar_adaptor): None
  (pfe): None
  (backbone_2d): BaseBEVBackbone(
    (blocks): ModuleList(
      (0): Sequential(
        (0): Identity()
        (1): Conv2d(256, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (2): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (3): ReLU()
        (4): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (5): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (6): ReLU()
        (7): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (8): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (9): ReLU()
        (10): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (11): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (12): ReLU()
        (13): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (14): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (15): ReLU()
        (16): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (17): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (18): ReLU()
      )
      (1): Sequential(
        (0): Identity()
        (1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (3): ReLU()
        (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (6): ReLU()
        (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (9): ReLU()
        (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (12): ReLU()
        (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (15): ReLU()
        (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (18): ReLU()
      )
    )
    (deblocks): ModuleList(
      (0): Sequential(
        (0): ConvTranspose2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (1): Sequential(
        (0): ConvTranspose2d(128, 128, kernel_size=(2, 2), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
        (2): ReLU()
      )
    )
  )
  (dense_head): CenterHead(
    (shared_conv): Sequential(
      (0): Conv2d(256, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
    )
    (heads_list): ModuleList(
      (0): SeparateHead(
        (center): Sequential(
          (0): Sequential(
            (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (2): ReLU()
          )
          (1): Conv2d(32, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        )
        (center_z): Sequential(
          (0): Sequential(
            (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (2): ReLU()
          )
          (1): Conv2d(32, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        )
        (dim): Sequential(
          (0): Sequential(
            (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (2): ReLU()
          )
          (1): Conv2d(32, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        )
        (rot): Sequential(
          (0): Sequential(
            (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (2): ReLU()
          )
          (1): Conv2d(32, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        )
        (hm): Sequential(
          (0): Sequential(
            (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (2): ReLU()
          )
          (1): Conv2d(32, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
        )
      )
    )
    (hm_loss_func): FocalLossCenterNet()
    (reg_loss_func): RegLossCenterNet()
  )
  (dense_head_aux): None
  (kd_adapt_block): None
  (point_head): None
  (roi_head): None
)
2023-01-09 18:02:42,205   INFO  **********************Start training waymo_models/cp-voxel/cp-voxel-s_sparsekd(default)**********************
epochs:   0%|          | 0/20 [00:00<?, ?it/s]
train:   0%|          | 0/199 [00:00<?, ?it/s]/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)

train:   1%|          | 1/199 [00:04<13:15,  4.02s/it]
epochs:   0%|          | 0/20 [00:04<?, ?it/s, kd_ls=3.22, kd_hm_ls=0.84, kd_loc_ls=2.38, kd_sort_ls=0.00, loss=14.7, lr=0.0003, d_t=(0.3), b_t=4.0(4.0)]
train:   1%|          | 2/199 [00:04<05:43,  1.74s/it, total_it=1]
epochs:   0%|          | 0/20 [00:04<?, ?it/s, kd_ls=3.98, kd_hm_ls=0.70, kd_loc_ls=3.28, kd_sort_ls=0.00, loss=21.6, lr=0.0003, d_t=(0.2), b_t=0.1(2.1)]
train:   2%|▏         | 3/199 [00:04<03:20,  1.02s/it, total_it=2]
epochs:   0%|          | 0/20 [00:04<?, ?it/s, kd_ls=2.69, kd_hm_ls=0.37, kd_loc_ls=2.32, kd_sort_ls=0.00, loss=15, lr=0.0003, d_t=(0.1), b_t=0.2(1.4)]
train:   2%|▏         | 4/199 [00:24<27:36,  8.49s/it, total_it=3]
epochs:   0%|          | 0/20 [00:24<?, ?it/s, kd_ls=3.33, kd_hm_ls=0.98, kd_loc_ls=2.35, kd_sort_ls=0.00, loss=23.1, lr=0.0003, d_t=(0.1), b_t=19.9(6.1)]
train:   3%|▎         | 5/199 [00:24<17:43,  5.48s/it, total_it=4]
epochs:   0%|          | 0/20 [00:24<?, ?it/s, kd_ls=3.13, kd_hm_ls=0.32, kd_loc_ls=2.81, kd_sort_ls=0.00, loss=18.5, lr=0.0003, d_t=(0.1), b_t=0.1(4.9)]
train:   3%|▎         | 6/199 [00:24<11:49,  3.68s/it, total_it=5]
epochs:   0%|          | 0/20 [00:24<?, ?it/s, kd_ls=2.80, kd_hm_ls=0.30, kd_loc_ls=2.49, kd_sort_ls=0.00, loss=12.2, lr=0.0003, d_t=(0.1), b_t=0.2(4.1)]
train:   4%|▎         | 7/199 [00:24<08:10,  2.55s/it, total_it=6]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=2.78, kd_hm_ls=0.36, kd_loc_ls=2.42, kd_sort_ls=0.00, loss=14.5, lr=0.0003, d_t=(0.0), b_t=0.2(3.5)]
train:   4%|▍         | 8/199 [00:24<05:41,  1.79s/it, total_it=7]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=3.76, kd_hm_ls=0.43, kd_loc_ls=3.33, kd_sort_ls=0.00, loss=11.9, lr=0.0003, d_t=(0.0), b_t=0.1(3.1)]
train:   5%|▍         | 9/199 [00:25<04:01,  1.27s/it, total_it=8]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=2.75, kd_hm_ls=0.24, kd_loc_ls=2.50, kd_sort_ls=0.00, loss=11.8, lr=0.0003, d_t=(0.0), b_t=0.1(2.8)]
train:   5%|▌         | 10/199 [00:25<02:58,  1.06it/s, total_it=9]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=3.13, kd_hm_ls=0.43, kd_loc_ls=2.70, kd_sort_ls=0.00, loss=13.7, lr=0.0003, d_t=(0.0), b_t=0.2(2.5)]
train:   6%|▌         | 11/199 [00:25<02:12,  1.42it/s, total_it=10]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=4.63, kd_hm_ls=1.55, kd_loc_ls=3.08, kd_sort_ls=0.00, loss=24, lr=0.0003, d_t=(0.0), b_t=0.2(2.3)]
train:   6%|▌         | 12/199 [00:25<01:37,  1.91it/s, total_it=11]
epochs:   0%|          | 0/20 [00:25<?, ?it/s, kd_ls=3.80, kd_hm_ls=0.72, kd_loc_ls=3.08, kd_sort_ls=0.00, loss=19.5, lr=0.0003, d_t=(0.0), b_t=0.1(2.1)]
train:   7%|▋         | 13/199 [00:25<01:21,  2.27it/s, total_it=12]
epochs:   0%|          | 0/20 [00:26<?, ?it/s, kd_ls=2.68, kd_hm_ls=0.33, kd_loc_ls=2.36, kd_sort_ls=0.00, loss=15.3, lr=0.0003, d_t=(0.0), b_t=0.2(2.0)]
Traceback (most recent call last):
  File "train.py", line 246, in <module>
    main()
  File "train.py", line 184, in main
    train_func(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/train_utils/train_kd_utils.py", line 136, in train_model_kd
    accumulated_iter = train_one_epoch(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/train_utils/train_kd_utils.py", line 56, in train_one_epoch
    loss, tb_dict, disp_dict = forward_func(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/utils/kd_utils/kd_forwad.py", line 127, in forward
    ret_dict, tb_dict, disp_dict = model(batch)
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/detectors/centerpoint.py", line 45, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/dense_heads/center_head.py", line 419, in forward
    target_dict = self.assign_targets(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/dense_heads/center_head.py", line 267, in assign_targets
    heatmap, ret_boxes, inds, mask = self.assign_target_of_single_head(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/dense_heads/center_head.py", line 205, in assign_target_of_single_head
    centernet_utils.draw_gaussian_to_heatmap(
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/model_utils/centernet_utils.py", line 49, in draw_gaussian_to_heatmap
    gaussian = gaussian2D((diameter, diameter), sigma=diameter / 6)
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/models/model_utils/centernet_utils.py", line 42, in gaussian2D
    h = np.exp(-(x * x + y * y) / (2 * sigma * sigma))
numpy.core._exceptions.MemoryError: Unable to allocate 458. GiB for an array with shape (247985, 247985) and data type float64

                                                                    2023-01-09 18:03:08,573   INFO  Deleting GT database from shared memory
Exception ignored in: <function DataBaseSampler.__del__ at 0x7fe64990d4c0>
Traceback (most recent call last):
  File "/home/chongqinghuang/code/light_weight/SparseKD/tools/../pcdet/datasets/augmentor/database_sampler.py", line 63, in __del__
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1446, in info
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1589, in _log
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1599, in handle
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 954, in handle
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1186, in emit
  File "/home/chongqinghuang/anaconda3/envs/pcdet/lib/python3.8/logging/__init__.py", line 1176, in _open
NameError: name 'open' is not defined

Process finished with exit code 1
jihanyang commented 1 year ago

It is an very strange error. Have you modified any config?