search

PaddlePaddle / PaddleOCR

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
https://paddlepaddle.github.io/PaddleOCR/
Apache License 2.0
43.9k stars 7.8k forks source link

怎么判断训练识别模型是否过拟合？ #4723

Closed chccc1994 closed 1 year ago

chccc1994 commented 2 years ago 
Global:
  use_gpu: true
  epoch_num: 72
  log_smooth_window: 20
  print_batch_step: 10
  save_model_dir: ./output/rec/r34_vd_none_bilstm_ctc
  save_epoch_step: 3
  # evaluation is run every 2000 iterations after the 0th iteration
  eval_batch_step: [0, 2000]
  cal_metric_during_train: true
  pretrained_model:
  checkpoints:
  save_inference_dir:
  use_visualdl: true
  infer_img: doc/imgs_words_en/word_10.png
  # for data or label process
  character_dict_path: 
  character_type: en
  max_text_length: 25
  infer_mode: false
  use_space_char: true
  save_res_path: ./output/rec/predicts_r34_vd_none_bilstm_ctc.txt

Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Cosine
    learning_rate: 0.0005
  regularizer:
    name: 'L2'
    factor: 0

Architecture:
  model_type: rec
  algorithm: CRNN
  Transform:
  Backbone:
    name: ResNet
    layers: 101
  Neck:
    name: SequenceEncoder
    encoder_type: rnn
    hidden_size: 256
  Head:
    name: CTCHead
    fc_decay: 0

Loss:
  name: CTCLoss

PostProcess:
  name: CTCLabelDecode

Metric:
  name: RecMetric
  main_indicator: acc

Train:
  dataset:
    name: SimpleDataSet
    data_dir: E:/Paddle/datasets/
    label_file_list: ["E:/Paddle/datasets/ICDAR2015/ICDAR2015_REC/rec_gt_train.txt","E:/Paddle/datasets/word_recognition/train.txt"]
    ratio_list: [1.0, 1.0]
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: false
      - CTCLabelEncode: # Class handling label
      - RecResizeImg:
          image_shape: [3, 32, 100]
      - KeepKeys:
          keep_keys: ['image', 'label', 'length'] # dataloader will return list in this order
  loader:
    shuffle: true
    batch_size_per_card: 64
    drop_last: true
    num_workers: 4

Eval:
  dataset:
    name: SimpleDataSet
    data_dir: E:/Paddle/datasets/ICDAR2015/ICDAR2015_REC/ch4_test_word_images_gt
    label_file_list: ["E:/Paddle/datasets/ICDAR2015/ICDAR2015_REC/rec_gt_test.txt"]
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: false
      - CTCLabelEncode: # Class handling label
      - RecResizeImg:
          image_shape: [3, 32, 100]
      - KeepKeys:
          keep_keys: ['image', 'label', 'length'] # dataloader will return list in this order
  loader:
    shuffle: false
    drop_last: false
    batch_size_per_card: 256
    num_workers: 4

**部分训练日志**

[2021/11/22 17:43:59] root INFO: epoch: [54/72], iter: 12620, lr: 0.000083, loss: 0.026682, acc: 0.992188, norm_edit_dis: 0.999023, reader_cost: 0.00332 s, batch_cost: 0.20981 s, samples: 448, ips: 213.52985
[2021/11/22 17:44:02] root INFO: epoch: [54/72], iter: 12630, lr: 0.000083, loss: 0.017296, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29536 s, samples: 640, ips: 216.68324
[2021/11/22 17:44:05] root INFO: epoch: [54/72], iter: 12640, lr: 0.000082, loss: 0.012721, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29636 s, samples: 640, ips: 215.95243
[2021/11/22 17:44:09] root INFO: epoch: [54/72], iter: 12650, lr: 0.000082, loss: 0.010732, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29794 s, samples: 640, ips: 214.81186
[2021/11/22 17:44:12] root INFO: epoch: [54/72], iter: 12660, lr: 0.000082, loss: 0.063091, acc: 0.984375, norm_edit_dis: 0.997396, reader_cost: 0.00049 s, batch_cost: 0.29965 s, samples: 640, ips: 213.57995
[2021/11/22 17:44:15] root INFO: epoch: [54/72], iter: 12670, lr: 0.000081, loss: 0.018359, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00020 s, batch_cost: 0.30165 s, samples: 640, ips: 212.16400
[2021/11/22 17:44:19] root INFO: epoch: [54/72], iter: 12680, lr: 0.000081, loss: 0.008360, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00029 s, batch_cost: 0.29717 s, samples: 640, ips: 215.36647
[2021/11/22 17:44:22] root INFO: epoch: [54/72], iter: 12690, lr: 0.000081, loss: 0.012984, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29574 s, samples: 640, ips: 216.40818
[2021/11/22 17:44:25] root INFO: epoch: [54/72], iter: 12700, lr: 0.000080, loss: 0.024742, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29524 s, samples: 640, ips: 216.77201
[2021/11/22 17:44:28] root INFO: epoch: [54/72], iter: 12710, lr: 0.000080, loss: 0.034740, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00029 s, batch_cost: 0.29604 s, samples: 640, ips: 216.18995
[2021/11/22 17:44:32] root INFO: epoch: [54/72], iter: 12720, lr: 0.000080, loss: 0.017820, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00068 s, batch_cost: 0.29616 s, samples: 640, ips: 216.09797
[2021/11/22 17:44:35] root INFO: epoch: [54/72], iter: 12730, lr: 0.000079, loss: 0.030346, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00020 s, batch_cost: 0.29354 s, samples: 640, ips: 218.02644
[2021/11/22 17:44:38] root INFO: epoch: [54/72], iter: 12740, lr: 0.000079, loss: 0.029520, acc: 0.992188, norm_edit_dis: 0.998698, reader_cost: 0.00059 s, batch_cost: 0.29623 s, samples: 640, ips: 216.04674
[2021/11/22 17:44:42] root INFO: epoch: [54/72], iter: 12750, lr: 0.000079, loss: 0.022735, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00020 s, batch_cost: 0.30108 s, samples: 640, ips: 212.56988
[2021/11/22 17:44:45] root INFO: epoch: [54/72], iter: 12760, lr: 0.000078, loss: 0.020928, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00029 s, batch_cost: 0.30250 s, samples: 640, ips: 211.57174
[2021/11/22 17:44:48] root INFO: epoch: [54/72], iter: 12770, lr: 0.000078, loss: 0.009819, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00049 s, batch_cost: 0.29762 s, samples: 640, ips: 215.04053
[2021/11/22 17:44:52] root INFO: epoch: [54/72], iter: 12780, lr: 0.000078, loss: 0.010094, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00039 s, batch_cost: 0.29722 s, samples: 640, ips: 215.32564
[2021/11/22 17:44:55] root INFO: epoch: [54/72], iter: 12790, lr: 0.000077, loss: 0.008973, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00049 s, batch_cost: 0.29622 s, samples: 640, ips: 216.05840
[2021/11/22 17:44:58] root INFO: epoch: [54/72], iter: 12800, lr: 0.000077, loss: 0.007852, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00029 s, batch_cost: 0.29585 s, samples: 640, ips: 216.32229
[2021/11/22 17:45:01] root INFO: epoch: [54/72], iter: 12810, lr: 0.000077, loss: 0.019468, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00059 s, batch_cost: 0.29619 s, samples: 640, ips: 216.07980
[2021/11/22 17:45:05] root INFO: epoch: [54/72], iter: 12820, lr: 0.000076, loss: 0.028947, acc: 1.000000, norm_edit_dis: 1.000000, reader_cost: 0.00029 s, batch_cost: 0.29493 s, samples: 640, ips: 217.00294
[2021/11/22 17:45:08] root INFO: epoch: [54/72], iter: 12830, lr: 0.000076, loss: 0.035733, acc: 0.992188, norm_edit_dis: 0.999132, reader_cost: 0.00039 s, batch_cost: 0.29683 s, samples: 640, ips: 215.60934
[2021/11/22 17:45:11] root INFO: epoch: [54/72], iter: 12840, lr: 0.000076, loss: 0.031900, acc: 0.984375, norm_edit_dis: 0.997396, reader_cost: 0.00049 s, batch_cost: 0.29939 s, samples: 640, ips: 213.76903
[2021/11/22 17:45:15] root INFO: epoch: [54/72], iter: 12850, lr: 0.000075, loss: 0.027874, acc: 0.984375, norm_edit_dis: 0.997396, reader_cost: 0.00029 s, batch_cost: 0.30134 s, samples: 640, ips: 212.38565

**模型评估**

[2021/11/22 19:45:53] root INFO: train with paddle 2.1.2 and device CUDAPlace(0)
[2021/11/22 19:45:53] root INFO: Initialize indexs of datasets:['E:/Paddle/datasets/ICDAR2015/ICDAR2015_REC/rec_gt_test.txt']
W1122 19:45:53.318128  3976 device_context.cc:404] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.3, Runtime API Version: 11.0
W1122 19:45:53.326764  3976 device_context.cc:422] device: 0, cuDNN Version: 8.0.
[2021/11/22 19:45:57] root INFO: resume from output/rec/r34_vd_none_bilstm_ctc/best_accuracy
[2021/11/22 19:45:57] root INFO: metric in ckpt ***************
[2021/11/22 19:45:57] root INFO: acc:0.46630859375
[2021/11/22 19:45:57] root INFO: norm_edit_dis:0.750480563273532
[2021/11/22 19:45:57] root INFO: fps:819.4431190055109
[2021/11/22 19:45:57] root INFO: best_epoch:59
[2021/11/22 19:45:57] root INFO: start_epoch:60

eval model::   0%|          | 0/9 [00:00<?, ?it/s]
eval model::  11%|█         | 1/9 [00:01<00:09,  1.19s/it]
eval model::  22%|██▏       | 2/9 [00:01<00:04,  1.46it/s]
eval model::  33%|███▎      | 3/9 [00:01<00:03,  1.93it/s]
eval model::  44%|████▍     | 4/9 [00:02<00:02,  2.25it/s]
eval model::  56%|█████▌    | 5/9 [00:02<00:01,  2.48it/s]
eval model::  67%|██████▋   | 6/9 [00:02<00:01,  2.66it/s]
eval model::  78%|███████▊  | 7/9 [00:03<00:00,  2.77it/s]
eval model::  89%|████████▉ | 8/9 [00:03<00:00,  2.86it/s]
eval model::  89%|████████▉ | 8/9 [00:03<00:00,  2.30it/s]
[2021/11/22 19:46:01] root INFO: metric eval ***************
[2021/11/22 19:46:01] root INFO: acc:0.46630859375
[2021/11/22 19:46:01] root INFO: norm_edit_dis:0.750480563273532
[2021/11/22 19:46:01] root INFO: fps:629.6481963813644
INFO 2021-11-22 19:46:02,460 launch.py:268] Local processes completed.

训练集大小1.5W张图像，ICDAR2015数据集和其他数据集。没有添加预训练模型。 训练epoch=72,在epoch=54时，acc=1，但是在模型评估eval时，acc:0.46630859375，是否是过拟合？怎么判断训练的识别模型是否过拟合？

LDOUBLEV commented 2 years ago

怎么判断训练的识别模型是否过拟合？

训练模型精度远大于eval模型精度的时候就是过拟合了

chccc1994 commented 2 years ago

训练集大小1.5W张图像，ICDAR2015数据集和其他数据集。没有添加预训练模型。 训练epoch=72,在epoch=54时，acc=1，但是在模型评估eval时，acc:0.46630859375，是否是过拟合？怎么判断训练的识别模型是否过拟合？

训练集大小1.5W张图像，ICDAR2015数据集和其他数据集。没有添加预训练模型。 训练epoch=72,在epoch=54时，acc=1，但是在模型评估eval时，acc:0.46630859375，是不是过拟合，现在怎么训练eval acc 都无法提高？

LDOUBLEV commented 2 years ago

多加一些数据增强

chccc1994 commented 2 years ago

多加一些数据增强

  1. 根据Q2.3.7：识别训练时，训练集精度已经到达90了，但验证集精度一直在70，涨不上去怎么办？

    Architecture:
Head:
fc_decay: 0.0005 # 由0 修改为0.0005
Train:
transforms:
  - RecAug: # 增加了增广
      use_tia: true
      aug_prob: 0.7

    训练完成： trian: acc=0.78 eval: acc=0.62 应该是正常范围吧？

  2. 但是未找到Q2.3.7中的l2_decay: 0.0005参数 含义？？

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.