检测模型finetune是不是太慢了

[kkpssr]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：ubuntu20
• 版本号/Version：Paddle：2.4 PaddleOCR：2.6
• 问题相关组件/Related components： 硬件为单卡3090，训练数据为254张高清发票(大概在1604x1000左右)，每张有7条左右标注，跑100个epoch，batch为8，大概需要11个小时左右，基于ch_PPOCRv3_student finetune 这个速度是不是有点离谱，总共才200多样本，是不是图片单张太大了?以下是训练配置 Global: debug: false use_gpu: true epoch_num: 100 log_smooth_window: 20 print_batch_step: 1 save_model_dir: ./output/ch_PP-OCR_V3_det/ save_epoch_step: 25 eval_batch_step:
  • 0
  • 100 cal_metric_during_train: false pretrained_model: null checkpoints: null save_inference_dir: null use_visualdl: false infer_img: doc/imgs_en/img_10.jpg save_res_path: ./checkpoints/det_db/predicts_db.txt distributed: true

Architecture: model_type: det algorithm: DB Transform: Backbone: name: MobileNetV3 scale: 0.5 model_name: large disable_se: True Neck: name: RSEFPN out_channels: 96 shortcut: True Head: name: DBHead k: 50

Loss: name: DBLoss balance_loss: true main_loss_type: DiceLoss alpha: 5 beta: 10 ohem_ratio: 3 Optimizer: name: Adam beta1: 0.9 beta2: 0.999 lr: name: Cosine learning_rate: 0.001 warmup_epoch: 2 regularizer: name: L2 factor: 5.0e-05 PostProcess: name: DBPostProcess thresh: 0.3 box_thresh: 0.6 max_candidates: 1000 unclip_ratio: 1.5 Metric: name: DetMetric main_indicator: hmean Train: dataset: name: SimpleDataSet data_dir: receipt/all label_file_list:

• invoice.txt ratio_list: [1.0] transforms:
  • DecodeImage: img_mode: BGR channel_first: false
  • DetLabelEncode: null
  • IaaAugment: augmenter_args:
    • type: Fliplr args: p: 0.5
    • type: Affine args: rotate:
      • -10
      • 10
    • type: Resize args: size:
      • 0.5
      • 0.25
  • EastRandomCropData: size:
    • 960
    • 960 max_tries: 50 keep_ratio: true
  • MakeBorderMap: shrink_ratio: 0.4 thresh_min: 0.3 thresh_max: 0.7
  • MakeShrinkMap: shrink_ratio: 0.4 min_text_size: 8
  • NormalizeImage: scale: 1./255. mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225 order: hwc
  • ToCHWImage: null
  • KeepKeys: keep_keys:
    • image
    • threshold_map
    • threshold_mask
    • shrink_map
    • shrink_mask loader: shuffle: true drop_last: false batch_size_per_card: 8 num_workers: 4 Eval: dataset: name: SimpleDataSet data_dir: receipt/all label_file_list:
• invoice.txt transforms:
  • DecodeImage: img_mode: BGR channel_first: false
  • DetLabelEncode: null
  • DetResizeForTest: null
  • NormalizeImage: scale: 1./255. mean:
    • 0.485
    • 0.456
    • 0.406 std:
    • 0.229
    • 0.224
    • 0.225 order: hwc
  • ToCHWImage: null
  • KeepKeys: keep_keys:
    • image
    • shape
    • polys
    • ignore_tags loader: shuffle: false drop_last: false batch_size_per_card: 1 num_workers: 2

[LDOUBLEV]

看你的训练配置，输入shape是960：

• EastRandomCropData: size:
• 960
• 960 max_tries

训练慢可能和硬件性能有关系，看下你的机器有没有其他进程在占资源，如果显存允许的话，可以加大batchsize； 另外，从日志中看下，batchcost的耗时，和reader的耗时

[flamebox]

我用3080ti训练也是，慢如果是显卡瓶颈还能接受，疑惑的是显卡负载一直上不去，断断续续只有30-40%这样的tdp功耗。batchsize已经调整至吃满显存了。怎样才能更高效率吃满显卡的算力？

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[zhangqiqi1228]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：ubuntu20
• 版本号/Version：Paddle：2.4 PaddleOCR：2.6

• 问题相关组件/Related components： 硬件为单卡3090，训练数据为254张高清发票(大概在1604x1000左右)，每张有7条左右标注，跑100个epoch，batch为8，大概需要11个小时左右，基于ch_PPOCRv3_student finetune 这个速度是不是有点离谱，总共才200多样本，是不是图片单张太大了?以下是训练配置 Global: debug: false use_gpu: true epoch_num: 100 log_smooth_window: 20 print_batch_step: 1 save_model_dir: ./output/ch_PP-OCR_V3_det/ save_epoch_step: 25 eval_batch_step:

  • 0
  • 100 cal_metric_during_train: false pretrained_model: null checkpoints: null save_inference_dir: null use_visualdl: false infer_img: doc/imgs_en/img_10.jpg save_res_path: ./checkpoints/det_db/predicts_db.txt distributed: true

Architecture: model_type: det algorithm: DB Transform: Backbone: name: MobileNetV3 scale: 0.5 model_name: large disable_se: True Neck: name: RSEFPN out_channels: 96 shortcut: True Head: name: DBHead k: 50

Loss: name: DBLoss balance_loss: true main_loss_type: DiceLoss alpha: 5 beta: 10 ohem_ratio: 3 Optimizer: name: Adam beta1: 0.9 beta2: 0.999 lr: name: Cosine learning_rate: 0.001 warmup_epoch: 2 regularizer: name: L2 factor: 5.0e-05 PostProcess: name: DBPostProcess thresh: 0.3 box_thresh: 0.6 max_candidates: 1000 unclip_ratio: 1.5 Metric: name: DetMetric main_indicator: hmean Train: dataset: name: SimpleDataSet data_dir: receipt/all label_file_list: - invoice.txt ratio_list: [1.0] transforms: - DecodeImage: img_mode: BGR channel_first: false - DetLabelEncode: null - IaaAugment: augmenter_args: - type: Fliplr args: p: 0.5 - type: Affine args: rotate: - -10 - 10 - type: Resize args: size: - 0.5 - 0.25 - EastRandomCropData: size: - 960 - 960 max_tries: 50 keep_ratio: true - MakeBorderMap: shrink_ratio: 0.4 thresh_min: 0.3 thresh_max: 0.7 - MakeShrinkMap: shrink_ratio: 0.4 min_text_size: 8 - NormalizeImage: scale: 1./255. mean: - 0.485 - 0.456 - 0.406 std: - 0.229 - 0.224 - 0.225 order: hwc - ToCHWImage: null - KeepKeys: keep_keys: - image - threshold_map - threshold_mask - shrink_map - shrink_mask loader: shuffle: true drop_last: false batch_size_per_card: 8 num_workers: 4 Eval: dataset: name: SimpleDataSet data_dir: receipt/all label_file_list: - invoice.txt transforms: - DecodeImage: img_mode: BGR channel_first: false - DetLabelEncode: null - DetResizeForTest: null - NormalizeImage: scale: 1./255. mean: - 0.485 - 0.456 - 0.406 std: - 0.229 - 0.224 - 0.225 order: hwc - ToCHWImage: null - KeepKeys: keep_keys: - image - shape - polys - ignore_tags loader: shuffle: false drop_last: false batch_size_per_card: 1 num_workers: 2

请问你使用 tools/train.py 这个脚本进行finetune的么？