ubuntu18.04 release0.4最新 GPU V100 自己数据集，四类

运行命令 CUDA_VISIBLE_DEVICES=1,2 python tools/train.py -c configs/ppyolo/ppyolo_grddc.yml --eval

错误

2020-09-21 15:48:39,420-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000500] in Optimizer will not take effect, and it will only be applied to other Parameters!
2020-09-21 15:48:46,059-INFO: places would be ommited when DataLoader is not iterable
W0921 15:48:46.160423  2666 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.2, Runtime API Version: 10.0
W0921 15:48:46.163522  2666 device_context.cc:260] device: 0, cuDNN Version: 8.0.
2020-09-21 15:48:48,595-WARNING: /root/.cache/paddle/weights/ResNet50_vd_ssld_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
/opt/conda/lib/python3.6/site-packages/paddle/fluid/io.py:1998: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.w_0 fc_0.b_0
  format(" ".join(unused_para_list)))
2020-09-21 15:48:52,352-INFO: places would be ommited when DataLoader is not iterable
I0921 15:48:55.301230  2666 build_strategy.cc:361] set enable_sequential_execution:1
W0921 15:48:56.606209  2666 fuse_all_reduce_op_pass.cc:74] Find all_reduce operators: 240. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 171.
W0921 15:49:18.407598  2739 init.cc:226] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0921 15:49:18.407640  2739 init.cc:228] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0921 15:49:18.407650  2739 init.cc:231] The detail failure signal is:

W0921 15:49:18.407660  2739 init.cc:234] *** Aborted at 1600703358 (unix time) try "date -d @1600703358" if you are using GNU date ***
W0921 15:49:18.409673  2739 init.cc:234] PC: @                0x0 (unknown)
W0921 15:49:18.409899  2739 init.cc:234] *** SIGSEGV (@0x0) received by PID 2666 (TID 0x7f43e0ffd700) from PID 0; stack trace: ***
W0921 15:49:18.411737  2739 init.cc:234]     @     0x7f44ed38f8a0 (unknown)
W0921 15:49:18.413237  2739 init.cc:234]     @                0x0 (unknown)
Segmentation fault (core dumped)

配置文件
architecture: YOLOv3
use_gpu: true
max_iters: 250000
log_smooth_window: 100
log_iter: 100
save_dir: output
snapshot_iter: 10000
metric: VOC
pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_vd_ssld_pretrained.tar
weights: output/ppyolo/model_final
num_classes: 4
use_fine_grained_loss: true
use_ema: true
ema_decay: 0.9998

YOLOv3:
  backbone: ResNet
  yolo_head: YOLOv3Head
  use_fine_grained_loss: true

ResNet:
  norm_type: sync_bn
  freeze_at: 0
  freeze_norm: false
  norm_decay: 0.
  depth: 50
  feature_maps: [3, 4, 5]
  variant: d
  dcn_v2_stages: [5]

YOLOv3Head:
  anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
  anchors: [[10, 13], [16, 30], [33, 23],
            [30, 61], [62, 45], [59, 119],
            [116, 90], [156, 198], [373, 326]]
  norm_decay: 0.
  coord_conv: true
  iou_aware: true
  iou_aware_factor: 0.4
  scale_x_y: 1.05
  spp: true
  yolo_loss: YOLOv3Loss
  nms: MatrixNMS
  drop_block: true

YOLOv3Loss:
  ignore_thresh: 0.7
  scale_x_y: 1.05
  label_smooth: false
  use_fine_grained_loss: true
  iou_loss: IouLoss
  iou_aware_loss: IouAwareLoss

IouLoss:
  loss_weight: 2.5
  max_height: 608
  max_width: 608

IouAwareLoss:
  loss_weight: 1.0
  max_height: 608
  max_width: 608

MatrixNMS:
    background_label: -1
    keep_top_k: 100
    normalized: false
    score_threshold: 0.01
    post_threshold: 0.01

LearningRate:
  base_lr: 0.0025
  schedulers:
  - !PiecewiseDecay
    gamma: 0.1
    milestones:
    - 150000
    - 200000
  - !LinearWarmup
    start_factor: 0.
    steps: 4000

OptimizerBuilder:
  optimizer:
    momentum: 0.9
    type: Momentum
  regularizer:
    factor: 0.0005
    type: L2

_READER_: 'ppyolo_reader_grddc.yml'

TrainReader:
  inputs_def:
    fields: ['image', 'gt_bbox', 'gt_class', 'gt_score']
    num_max_boxes: 50
  dataset:
    !VOCDataSet
      anno_path: trainval.txt  
      dataset_dir: dataset/voc
      use_default_label: false   
      with_background: false  
  sample_transforms:
    - !DecodeImage
      to_rgb: True
      with_mixup: True
    - !MixupImage
      alpha: 1.5
      beta: 1.5
    - !ColorDistort {}
    - !RandomExpand
      fill_value: [123.675, 116.28, 103.53]
    - !RandomCrop {}
    - !RandomFlipImage
      is_normalized: false
    - !NormalizeBox {}
    - !PadBox
      num_max_boxes: 50
    - !BboxXYXY2XYWH {}
  batch_transforms:
  - !RandomShape
    sizes: [320, 352, 384, 416, 448, 480, 512, 544, 576, 608]
    random_inter: True
  - !NormalizeImage
    mean: [0.485, 0.456, 0.406]
    std: [0.229, 0.224, 0.225]
    is_scale: True
    is_channel_first: false
  - !Permute
    to_bgr: false
    channel_first: True
  # Gt2YoloTarget is only used when use_fine_grained_loss set as true,
  # this operator will be deleted automatically if use_fine_grained_loss
  # is set as false
  - !Gt2YoloTarget
    anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
    anchors: [[10, 13], [16, 30], [33, 23],
              [30, 61], [62, 45], [59, 119],
              [116, 90], [156, 198], [373, 326]]
    downsample_ratios: [32, 16, 8]
  batch_size: 12
  shuffle: true
  mixup_epoch: 25000
  drop_last: true
  worker_num: 1
  bufsize: 4
  use_process: false

EvalReader:
  inputs_def:
    fields: ['image', 'im_size', 'im_id']
    num_max_boxes: 50
  dataset:
    !VOCDataSet
      anno_path: test.txt
      dataset_dir: dataset/voc
      use_default_label: false
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !PadBox
      num_max_boxes: 50
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 8
  drop_empty: false
  worker_num: 1
  bufsize: 4

TestReader:
  inputs_def:
    image_shape: [3, 608, 608]
    fields: ['image', 'im_size', 'im_id']
  dataset:
    !ImageFolder
      anno_path: annotations/instances_val2017.json
      with_background: false
  sample_transforms:
    - !DecodeImage
      to_rgb: True
    - !ResizeImage
      target_size: 608
      interp: 2
    - !NormalizeImage
      mean: [0.485, 0.456, 0.406]
      std: [0.229, 0.224, 0.225]
      is_scale: True
      is_channel_first: false
    - !Permute
      to_bgr: false
      channel_first: True
  batch_size: 1

此外数据集存在空gt的情况，存在没有difficult标签的情况，因此更改如下文件：

PaddleDetection/ppdet/data/source/voc.py line145

#_difficult = int(obj.find('difficult').text)
_difficult = 0

PaddleDetection/ppdet/data/transform/operators.py Line1272

    gt_class1 = sample['gt_class'].reshape((-1, 1))
    gt_class2 = sample['mixup']['gt_class'].reshape((-1, 1))
    gt_class = np.concatenate((gt_class1, gt_class2), axis=0)

    gt_score1 = sample['gt_score'].reshape((-1, 1))
    gt_score2 = sample['mixup']['gt_score'].reshape((-1, 1))
    gt_score = np.concatenate(
        (gt_score1 * factor, gt_score2 * (1. - factor)), axis=0)

    is_crowd1 = sample['is_crowd'].reshape((-1, 1))
    is_crowd2 = sample['mixup']['is_crowd'].reshape((-1, 1))
    is_crowd = np.concatenate((is_crowd1, is_crowd2), axis=0)

YOLO系列本身支持数据gt为空的情况。错误不像是代码改动导致的。但错误信息不够详细，目前还没看出什么原因。不知道还有没有其他信息，比如单卡能否跑起来？其他的log等。或 export GLOG_vmoudle=operator=4 看下有没有其他log。

YOLO系列本身支持数据gt为空的情况。错误不像是代码改动导致的。但错误信息不够详细，目前还没看出什么原因。不知道还有没有其他信息，比如单卡能否跑起来？其他的log等。或 export GLOG_vmoudle=operator=4 看下有没有其他log。

单卡也不行，完全不知道该从哪判断，找不到其他log， export GLOG_vmoudle=operator=4运行后也没有其他log。自己的数据集拿faster系列能跑起来

训练faster_rcnn_dcn_x101_vd_64x4d_fpn_1x和efficentnet都没问题，但是ppyolo不行，CBResNet200-vd-FPN-Nonlocal也不行

CUDA_VISIBLE_DEVICES=0 python tools/train.py -c configs/dcn/cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms_grddc.yml --eval 2020-09-24 14:22:16,925-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000100] in Optimizer will not take effect, and it will only be applied to other Parameters! 2020-09-24 14:22:26,767-INFO: places would be ommited when DataLoader is not iterable W0924 14:22:26.894659 9687 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.2, Runtime API Version: 10.0 W0924 14:22:26.897580 9687 device_context.cc:260] device: 0, cuDNN Version: 8.0. 2020-09-24 14:22:29,009-WARNING: /root/.cache/paddle/weights/ResNet200_vd_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ] /opt/conda/envs/paddle/lib/python3.7/site-packages/paddle/fluid/io.py:1998: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.w_0 fc_0.b_0 format(" ".join(unused_para_list))) 2020-09-24 14:22:30,154-INFO: places would be ommited when DataLoader is not iterable 2020-09-24 14:22:33,640-INFO: iter: 0, lr: 0.001000, 'loss_cls_0': '1.689152', 'loss_loc_0': '0.000000', 'loss_cls_1': '0.870103', 'loss_loc_1': '0.000002', 'loss_cls_2': '0.379178', 'loss_loc_2': '0.000001', 'loss_rpn_cls': '0.687545', 'loss_rpn_bbox': '0.073837', 'loss': '3.699818', time: 0.000, eta: 0:00:00 W0924 14:22:34.125052 9844 init.cc:226] Warning: PaddlePaddle catches a failure signal, it may not work properly W0924 14:22:34.125079 9844 init.cc:228] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0924 14:22:34.125083 9844 init.cc:231] The detail failure signal is:

W0924 14:22:34.125087 9844 init.cc:234] Aborted at 1600957354 (unix time) try "date -d @1600957354" if you are using GNU date W0924 14:22:34.126740 9844 init.cc:234] PC: @ 0x0 (unknown) W0924 14:22:34.126876 9844 init.cc:234] SIGSEGV (@0x0) received by PID 9687 (TID 0x7f7e6e364700) from PID 0; stack trace: W0924 14:22:34.128216 9844 init.cc:234] @ 0x7f7ef25098a0 (unknown) W0924 14:22:34.129441 9844 init.cc:234] @ 0x0 (unknown)

config architecture: CascadeRCNNClsAware max_iters: 10000 snapshot_iter: 1000 use_gpu: true log_smooth_window: 20 save_dir: output pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet200_vd_pretrained.tar weights: output/cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms/model_final metric: VOC num_classes: 5

CascadeRCNNClsAware: backbone: ResNet fpn: FPN rpn_head: FPNRPNHead roi_extractor: FPNRoIAlign bbox_head: CascadeBBoxHead bbox_assigner: CascadeBBoxAssigner

ResNet: norm_type: bn depth: 200 feature_maps: [2, 3, 4, 5] freeze_at: 2 variant: d dcn_v2_stages: [3, 4, 5] nonlocal_stages: [4]

FPN: min_level: 2 max_level: 6 num_chan: 256 spatial_scale: [0.03125, 0.0625, 0.125, 0.25]

FPNRPNHead: anchor_generator: anchor_sizes: [32, 64, 128, 256, 512] aspect_ratios: [0.5, 1.0, 2.0] stride: [16.0, 16.0] variance: [1.0, 1.0, 1.0, 1.0] anchor_start_size: 32 min_level: 2 max_level: 6 num_chan: 256 rpn_target_assign: rpn_batch_size_per_im: 256 rpn_fg_fraction: 0.5 rpn_positive_overlap: 0.7 rpn_negative_overlap: 0.3 rpn_straddle_thresh: 0.0 train_proposal: min_size: 0.0 nms_thresh: 0.7 pre_nms_top_n: 2000 post_nms_top_n: 2000 test_proposal: min_size: 0.0 nms_thresh: 0.7 pre_nms_top_n: 1000 post_nms_top_n: 1000

FPNRoIAlign: canconical_level: 4 canonical_size: 224 min_level: 2 max_level: 5 box_resolution: 14 sampling_ratio: 2

CascadeBBoxAssigner: batch_size_per_im: 512 bbox_reg_weights: [10, 20, 30] bg_thresh_lo: [0.0, 0.0, 0.0] bg_thresh_hi: [0.5, 0.6, 0.7] fg_thresh: [0.5, 0.6, 0.7] fg_fraction: 0.25 class_aware: True

CascadeBBoxHead: head: CascadeTwoFCHead nms: MultiClassSoftNMS

CascadeTwoFCHead: mlp_dim: 1024

MultiClassSoftNMS: score_threshold: 0.01 keep_top_k: 300 softnms_sigma: 0.5

LearningRate: base_lr: 0.01 schedulers:

!PiecewiseDecay gamma: 0.1 milestones: [340000, 440000]
!LinearWarmup start_factor: 0.1 steps: 1000

OptimizerBuilder: optimizer: momentum: 0.9 type: Momentum regularizer: factor: 0.0001 type: L2

TrainReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'gt_bbox', 'gt_class', 'is_crowd'] dataset: !VOCDataSet dataset_dir: dataset/voc anno_path: trainval1.txt use_default_label: false sample_transforms:

!DecodeImage to_rgb: true
!RandomFlipImage prob: 0.5
!NormalizeImage is_channel_first: false is_scale: True mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
!ResizeImage interp: 1 target_size: [416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928, 960, 992, 1024, 1056, 1088, 1120, 1152, 1184, 1216, 1248, 1280, 1312, 1344, 1376, 1408] max_size: 1800 use_cv2: true
!Permute to_bgr: false batch_transforms:
!PadBatch pad_to_stride: 32 batch_size: 1 shuffle: false drop_last: false worker_num: 2

EvalReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'im_shape'] dataset: !VOCDataSet dataset_dir: dataset/voc anno_path: test.txt use_default_label: false sample_transforms:

!DecodeImage to_rgb: True with_mixup: False
!NormalizeImage is_channel_first: false is_scale: True mean:
- 0.485
- 0.456
- 0.406 std:
- 0.229
- 0.224
- 0.225
!ResizeImage interp: 1 target_size:
- 1200 max_size: 2000 use_cv2: true
!Permute to_bgr: false batch_transforms:
!PadBatch pad_to_stride: 32 batch_size: 1 worker_num: 2 drop_empty: false

TestReader: inputs_def: fields: ['image', 'im_info', 'im_id', 'im_shape'] dataset: !ImageFolder anno_path: dataset/voc/label_list.txt use_default_label: false

with_background: false

sample_transforms:

!DecodeImage to_rgb: true with_mixup: false
!NormalizeImage is_channel_first: false is_scale: true mean: [0.485,0.456,0.406] std: [0.229, 0.224,0.225]
!ResizeImage interp: 1 max_size: 1333 target_size: 800 use_cv2: true
!Permute channel_first: true to_bgr: false batch_transforms:
!PadBatch pad_to_stride: 32 use_padded_im_info: true batch_size: 1 worker_num: 2

该怎么找问题啊，没有详细log，debug一般是run处出错

@lurenlym ，你试试将数据增强中的Mixup方法删除，好像就可以我碰到同样的问题，删除之后，就可以了，但是不知道为什么？

PaddlePaddle / PaddleDetection

PPYOLO训练自己数据集出错 #1463

with_background: false