(Fatal) Blocking queue is killed because the data reader raises an exception.

[liu-0825]

跑caddn报错，没有做任何改动 ------------Environment Information------------- platform: Linux-4.14.0_1-0-0-36-x86_64-with-Ubuntu-18.04-bionic gcc (GCC) 8.2.0 Python - 3.7.10 (default, Feb 20 2021, 21:17:23) [GCC 7.5.0]

Science Toolkits: cv2 - 4.6.0 numpy - 1.21.6 numba - 0.56.4 pandas - 1.3.5 pillow - 8.3.2 skimage - 0.19.3

PaddlePaddle: paddle - 2.4.0 paddle3d - 0.5.0 paddleseg - 2.6.0 FLAGS_cudnn_deterministic - Not set. FLAGS_cudnn_exhaustive_search - Not set.

CUDA: cudnn - None nvcc - Build cuda_11.0_bu.TC445_37.28845127_0

GPUs: Not Found.

2022-11-24 14:56:07,165 - INFO - ---------------Config Information--------------- batch_size: 4 iters: 74240 lr_scheduler: div_factor: 10 lr_max: 0.001 moms:

• 0.95
• 0.85 pct_start: 0.4 total_step: 74240 type: OneCycle model: backbone_3d: layers: 101 return_idx:
  • 0
  • 3 type: ResNet bev_cfg: input_channels: 64 layer_nums:
  • 10
  • 10
  • 10 layer_strides:
  • 2
  • 2
  • 2 num_filters:
  • 64
  • 128
  • 256 num_upsample_filters:
  • 128
  • 128
  • 128 upsample_strides:
  • 1
  • 2
  • 4 class_head: align_corners: false aspp_out_channels: 256 aspp_ratios:
  • 1
  • 12
  • 24
  • 36 backbone_channels: 2048 backbone_indices:
  • 1 num_classes: 81 type: DeepLabV3Head dense_head: anchor_generator_cfg:
  • align_center: false anchor_bottom_heights:
    • -1.78 anchor_rotations:
    • 0
    • 1.57 anchor_sizes:
    • • 3.9
    • 1.6
    • 1.56 class_name: Car feature_map_stride: 2 matched_threshold: 0.6 unmatched_threshold: 0.45
  • align_center: false anchor_bottom_heights:
    • -0.6 anchor_rotations:
    • 0
    • 1.57 anchor_sizes:
    • • 0.8
    • 0.6
    • 1.73 class_name: Pedestrian feature_map_stride: 2 matched_threshold: 0.5 unmatched_threshold: 0.35
  • align_center: false anchor_bottom_heights:
    • -0.6 anchor_rotations:
    • 0
    • 1.57 anchor_sizes:
    • • 1.76
    • 0.6
    • 1.73 class_name: Cyclist feature_map_stride: 2 matched_threshold: 0.5 unmatched_threshold: 0.35 anchor_target_cfg: match_height: false norm_by_num_examples: false pos_fraction: -1.0 sample_size: 512 class_names:
  • Car
  • Pedestrian
  • Cyclist grid_size:
  • 280
  • 376
  • 25.0 input_channels: 384 loss_weights: cls_weight: 1.0 code_weights:
    • 1.0
    • 1.0
    • 1.0
    • 1.0
    • 1.0
    • 1.0
    • 1.0 dir_weight: 0.2 loc_weight: 2.0 model_cfg: class_agnostic: false dir_limit_offset: 0.0 dir_offset: 0.78539 use_direction_classifier: true num_dir_bins: 2 point_cloud_range:
  • 2
  • -30.08
  • -3.0
  • 46.8
  • 30.08
  • 1.0 predict_boxes_when_training: false type: AnchorHeadSingle disc_cfg: depth_max: 46.8 depth_min: 2.0 mode: LID num_bins: 80 f2v_cfg: grid_size:
  • 280
  • 376
  • 25.0 pc_range:
  • 2
  • -30.08
  • -3.0
  • 46.8
  • 30.08
  • 1.0 sample_cfg: mode: bilinear padding_mode: zeros ffe_cfg: channel_reduce_cfg: bias_attr: false in_channels: 256 kernel_size: 1 out_channels: 64 padding: 0 stride: 1 ddn_loss: alpha: 0.25 beta: 2.0 bg_weight: 1 fg_weight: 13 weight: 3.0 downsample_factor: 4 map_to_bev_cfg: bias_attr: false in_channels: 1600 kernel_size: 1 out_channels: 64 padding: 0 stride: 1 post_process_cfg: eval_metric: kitti nms_config: nms_post_maxsize: 500 nms_pre_maxsize: 4096 nms_thresh: 0.01 recall_thresh_list:
  • 0.3
  • 0.5
  • 0.7 score_thresh: 0.1 pretrained: https://bj.bcebos.com/paddle3d/pretrained/deeplabv3_resnet101.pdparams type: CADDN optimizer: beta1: 0.95 beta2: 0.99 clip_grad_by_norm: 10.0 type: AdamWOnecycle weight_decay: 0.01 sync_bn: true train_dataset: class_names:
• Car
• Pedestrian
• Cyclist dataset_root: data/kitti depth_downsample_factor: 4 mode: train point_cloud_range:
• 2
• -30.08
• -3.0
• 46.8
• 30.08
• 1.0 type: KittiDepthDataset voxel_size:
• 0.16
• 0.16
• 0.16 val_dataset: class_names:
• Car
• Pedestrian
• Cyclist dataset_root: data/kitti depth_downsample_factor: 4 mode: val point_cloud_range:
• 2
• -30.08
• -3.0
• 46.8
• 30.08
• 1.0 type: KittiDepthDataset voxel_size:
• 0.16
• 0.16

• 0.16

  2022-11-24 14:56:10,256 - WARNING - No custom op grid_sample_3d found, try JIT build Compiling user custom op, it will cost a few seconds..... 2022-11-24 14:56:12,770 - INFO - grid_sample_3d builded success! 2022-11-24 14:56:12,772 - WARNING - There is a file with the same name locally, we directly load the local file 2022-11-24 14:56:13,348 - WARNING - class_head.cls.weight is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - class_head.cls.bias is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - ffe.channel_reduce._conv.weight is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - ffe.channel_reduce._batch_norm.weight is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - ffe.channel_reduce._batch_norm.bias is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - ffe.channel_reduce._batch_norm._mean is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - ffe.channel_reduce._batch_norm._variance is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - map_to_bev._conv.weight is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - map_to_bev._batch_norm.weight is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - map_to_bev._batch_norm.bias is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - map_to_bev._batch_norm._mean is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - map_to_bev._batch_norm._variance is not in pretrained model 2022-11-24 14:56:13,349 - WARNING - backbone_2d.blocks.level_0.1.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.2.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.2.bias is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.2._mean is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.2._variance is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.4.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.5.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.5.bias is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.5._mean is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.5._variance is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.7.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.8.weight is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.8.bias is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.8._mean is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.8._variance is not in pretrained model 2022-11-24 14:56:13,350 - WARNING - backbone_2d.blocks.level_0.10.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.11.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.11.bias is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.11._mean is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.11._variance is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.13.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.14.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.14.bias is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.14._mean is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.14._variance is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.16.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.17.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.17.bias is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.17._mean is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.17._variance is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.19.weight is not in pretrained model 2022-11-24 14:56:13,351 - WARNING - backbone_2d.blocks.level_0.20.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.20.bias is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.20._mean is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.20._variance is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.22.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.23.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.23.bias is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.23._mean is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.23._variance is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.25.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.26.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.26.bias is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.26._mean is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.26._variance is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.28.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.29.weight is not in pretrained model 2022-11-24 14:56:13,352 - WARNING - backbone_2d.blocks.level_0.29.bias is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.29._mean is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.29._variance is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.31.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.32.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.32.bias is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.32._mean is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_0.32._variance is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.1.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.2.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.2.bias is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.2._mean is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.2._variance is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.4.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.5.weight is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.5.bias is not in pretrained model 2022-11-24 14:56:13,353 - WARNING - backbone_2d.blocks.level_1.5._mean is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.5._variance is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.7.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.8.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.8.bias is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.8._mean is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.8._variance is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.10.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.11.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.11.bias is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.11._mean is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.11._variance is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.13.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.14.weight is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.14.bias is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.14._mean is not in pretrained model 2022-11-24 14:56:13,354 - WARNING - backbone_2d.blocks.level_1.14._variance is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.16.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.17.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.17.bias is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.17._mean is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.17._variance is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.19.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.20.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.20.bias is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.20._mean is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.20._variance is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.22.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.23.weight is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.23.bias is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.23._mean is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.23._variance is not in pretrained model 2022-11-24 14:56:13,355 - WARNING - backbone_2d.blocks.level_1.25.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.26.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.26.bias is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.26._mean is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.26._variance is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.28.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.29.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.29.bias is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.29._mean is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.29._variance is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.31.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.32.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.32.bias is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.32._mean is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_1.32._variance is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_2.1.weight is not in pretrained model 2022-11-24 14:56:13,356 - WARNING - backbone_2d.blocks.level_2.2.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.2.bias is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.2._mean is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.2._variance is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.4.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.5.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.5.bias is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.5._mean is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.5._variance is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.7.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.8.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.8.bias is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.8._mean is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.8._variance is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.10.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.11.weight is not in pretrained model 2022-11-24 14:56:13,357 - WARNING - backbone_2d.blocks.level_2.11.bias is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.11._mean is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.11._variance is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.13.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.14.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.14.bias is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.14._mean is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.14._variance is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.16.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.17.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.17.bias is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.17._mean is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.17._variance is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.19.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.20.weight is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.20.bias is not in pretrained model 2022-11-24 14:56:13,358 - WARNING - backbone_2d.blocks.level_2.20._mean is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.20._variance is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.22.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.23.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.23.bias is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.23._mean is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.23._variance is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.25.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.26.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.26.bias is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.26._mean is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.26._variance is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.28.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.29.weight is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.29.bias is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.29._mean is not in pretrained model 2022-11-24 14:56:13,359 - WARNING - backbone_2d.blocks.level_2.29._variance is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.blocks.level_2.31.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.blocks.level_2.32.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.blocks.level_2.32.bias is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.blocks.level_2.32._mean is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.blocks.level_2.32._variance is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_0.0.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_0.1.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_0.1.bias is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_0.1._mean is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_0.1._variance is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_1.0.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_1.1.weight is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_1.1.bias is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_1.1._mean is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_1.1._variance is not in pretrained model 2022-11-24 14:56:13,360 - WARNING - backbone_2d.deblocks.level_2.0.weight is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - backbone_2d.deblocks.level_2.1.weight is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - backbone_2d.deblocks.level_2.1.bias is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - backbone_2d.deblocks.level_2.1._mean is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - backbone_2d.deblocks.level_2.1._variance is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_cls.weight is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_cls.bias is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_box.weight is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_box.bias is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_dir_cls.weight is not in pretrained model 2022-11-24 14:56:13,361 - WARNING - dense_head.conv_dir_cls.bias is not in pretrained model 2022-11-24 14:56:13,626 - INFO - There are 555/753 variables loaded into CADDN. Traceback (most recent call last): File "tools/train.py", line 189, in main(args) File "tools/train.py", line 184, in main trainer.train()Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 536, in _thread_loop batch = self._get_data() File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 674, in _get_data batch.reraise() File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/worker.py", line 172, in reraise raise self.exc_type(msg) AssertionError: DataLoader worker(0) caught AssertionError with message: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/worker.py", line 339, in _worker_loop batch = fetcher.fetch(indices) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/fetcher.py", line 125, in fetch data.append(self.dataset[idx]) File "/root/paddlejob/workspace/liujingkai/Paddle3D-develop/paddle3d/datasets/kitti/kitti_depth_det.py", line 458, in getitem data_dict = self.update_data(data_dict=input_dict) File "/root/paddlejob/workspace/liujingkai/Paddle3D-develop/paddle3d/datasets/kitti/kitti_depth_det.py", line 295, in update_data data_dict['depth_maps'] = self.get_depth_map(data_dict["frame_id"]) File "/root/paddlejob/workspace/liujingkai/Paddle3D-develop/paddle3d/datasets/kitti/kitti_depth_det.py", line 143, in get_depth_map assert os.path.exists(depth_file) AssertionError

  File "/root/paddlejob/workspace/liujingkai/Paddle3D-develop/paddle3d/apis/trainer.py", line 217, in train for sample in self.train_dataloader: File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dataloader/dataloader_iter.py", line 745, in next self._reader.read_nextlist()[0]) SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at /paddle/paddle/fluid/operators/reader/blocking_queue.h:175)

[wangna11BD]

看报错是depth_file不存在，可以检查一下自己的数据集～

[nepeplwu]

该issue较长时间无反馈，先关闭，如有问题请重新打开或者建立新的issue