search

PaddlePaddle / PaddleOCR

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)
https://paddlepaddle.github.io/PaddleOCR/
Apache License 2.0
42.47k stars 7.66k forks source link

请教-识别模型训练慢的问题 #3230

Closed wa3926 closed 3 years ago

wa3926 commented 3 years ago

paddle 版本 2.0.2 单卡V100 32G 自己准备数据集训练手写体识别 数据500W 使用 rec_chinese_lite_train_v2.0.yml 配置文件改的 训练过程中发现训练很慢
后来将 print_batch_step 改成1 发现 始终有进程 加载数据很慢 服务器上所有程序都被我关了 batch_size_per_card 改成256 也还是慢 刚开始还是有规律的 训练一会之后 reader_cost 感觉就乱了 不太稳定
后来我将num_workers 改成了 24 速度提上去了一点 截图在最下面 看了下top自己使用也就18个进程左右 磁盘不是固态的 cpu 是 Silver 4214 CPU @ 2.20GHz 500W 训练大概10个小时第一轮 不知道是不是cpu原因 就大佬指点下 还可能是哪些原因！

[2021/07/01 09:49:23] root INFO: Architecture : [2021/07/01 09:49:23] root INFO: Backbone : [2021/07/01 09:49:23] root INFO: model_name : small [2021/07/01 09:49:23] root INFO: name : MobileNetV3 [2021/07/01 09:49:23] root INFO: scale : 0.5 [2021/07/01 09:49:23] root INFO: small_stride : [1, 2, 2, 2] [2021/07/01 09:49:23] root INFO: Head : [2021/07/01 09:49:23] root INFO: fc_decay : 1e-05 [2021/07/01 09:49:23] root INFO: name : CTCHead [2021/07/01 09:49:23] root INFO: Neck : [2021/07/01 09:49:23] root INFO: encoder_type : rnn [2021/07/01 09:49:23] root INFO: hidden_size : 48 [2021/07/01 09:49:23] root INFO: name : SequenceEncoder [2021/07/01 09:49:23] root INFO: Transform : None [2021/07/01 09:49:23] root INFO: algorithm : CRNN [2021/07/01 09:49:23] root INFO: model_type : rec [2021/07/01 09:49:23] root INFO: Eval : [2021/07/01 09:49:23] root INFO: dataset : [2021/07/01 09:49:23] root INFO: data_dir : /paddle/data/handwrite/train_data/ [2021/07/01 09:49:23] root INFO: label_file_list : ['/paddle/data/handwrite/train_data/5660/test.txt'] [2021/07/01 09:49:23] root INFO: name : SimpleDataSet [2021/07/01 09:49:23] root INFO: transforms : [2021/07/01 09:49:23] root INFO: DecodeImage : [2021/07/01 09:49:23] root INFO: channel_first : False [2021/07/01 09:49:23] root INFO: img_mode : BGR [2021/07/01 09:49:23] root INFO: CTCLabelEncode : None [2021/07/01 09:49:23] root INFO: RecResizeImg : [2021/07/01 09:49:23] root INFO: image_shape : [3, 32, 320] [2021/07/01 09:49:23] root INFO: KeepKeys : [2021/07/01 09:49:23] root INFO: keep_keys : ['image', 'label', 'length'] [2021/07/01 09:49:23] root INFO: loader : [2021/07/01 09:49:23] root INFO: batch_size_per_card : 512 [2021/07/01 09:49:23] root INFO: drop_last : False [2021/07/01 09:49:23] root INFO: num_workers : 8 [2021/07/01 09:49:23] root INFO: shuffle : False [2021/07/01 09:49:23] root INFO: Global : [2021/07/01 09:49:23] root INFO: cal_metric_during_train : True [2021/07/01 09:49:23] root INFO: character_dict_path : /paddle/data/handwrite/train_data/5660/5660dict.txt [2021/07/01 09:49:23] root INFO: character_type : ch [2021/07/01 09:49:23] root INFO: checkpoints : None [2021/07/01 09:49:23] root INFO: debug : False [2021/07/01 09:49:23] root INFO: distributed : False [2021/07/01 09:49:23] root INFO: epoch_num : 500 [2021/07/01 09:49:23] root INFO: eval_batch_step : [180000, 30000] [2021/07/01 09:49:23] root INFO: infer_img : doc/imgs_words/ch/word_1.jpg [2021/07/01 09:49:23] root INFO: infer_mode : False [2021/07/01 09:49:23] root INFO: log_smooth_window : 20 [2021/07/01 09:49:23] root INFO: max_text_length : 25 [2021/07/01 09:49:23] root INFO: pretrained_model : /paddle/PaddleOCR/output/rec_chinese_lite_v2.0_myhandwrite_5660_49W/best_accuracy [2021/07/01 09:49:23] root INFO: print_batch_step : 1 [2021/07/01 09:49:23] root INFO: save_epoch_step : 3 [2021/07/01 09:49:23] root INFO: save_inference_dir : None [2021/07/01 09:49:23] root INFO: save_model_dir : ./output/rec_chinese_lite_v2.0_myhandwrite_5660_500W [2021/07/01 09:49:23] root INFO: save_res_path : ./output/rec/predicts_chinese_lite_v2.0_myhandwrite_5660_500W.txt [2021/07/01 09:49:23] root INFO: use_gpu : True [2021/07/01 09:49:23] root INFO: use_space_char : False [2021/07/01 09:49:23] root INFO: use_visualdl : False [2021/07/01 09:49:23] root INFO: Loss : [2021/07/01 09:49:23] root INFO: name : CTCLoss [2021/07/01 09:49:23] root INFO: Metric : [2021/07/01 09:49:23] root INFO: main_indicator : acc [2021/07/01 09:49:23] root INFO: name : RecMetric [2021/07/01 09:49:23] root INFO: Optimizer : [2021/07/01 09:49:23] root INFO: beta1 : 0.9 [2021/07/01 09:49:23] root INFO: beta2 : 0.999 [2021/07/01 09:49:23] root INFO: lr : [2021/07/01 09:49:23] root INFO: learning_rate : 0.001 [2021/07/01 09:49:23] root INFO: name : Cosine [2021/07/01 09:49:23] root INFO: name : Adam [2021/07/01 09:49:23] root INFO: regularizer : [2021/07/01 09:49:23] root INFO: factor : 1e-05 [2021/07/01 09:49:23] root INFO: name : L2 [2021/07/01 09:49:23] root INFO: PostProcess : [2021/07/01 09:49:23] root INFO: name : CTCLabelDecode [2021/07/01 09:49:23] root INFO: Train : [2021/07/01 09:49:23] root INFO: dataset : [2021/07/01 09:49:23] root INFO: data_dir : /paddle/data/handwrite/train_data/ [2021/07/01 09:49:23] root INFO: label_file_list : ['/paddle/data/handwrite/train_data/5660/train.txt'] [2021/07/01 09:49:23] root INFO: name : SimpleDataSet [2021/07/01 09:49:23] root INFO: transforms : [2021/07/01 09:49:23] root INFO: DecodeImage : [2021/07/01 09:49:23] root INFO: channel_first : False [2021/07/01 09:49:23] root INFO: img_mode : BGR [2021/07/01 09:49:23] root INFO: RecAug : None [2021/07/01 09:49:23] root INFO: CTCLabelEncode : None [2021/07/01 09:49:23] root INFO: RecResizeImg : [2021/07/01 09:49:23] root INFO: image_shape : [3, 32, 320] [2021/07/01 09:49:23] root INFO: KeepKeys : [2021/07/01 09:49:23] root INFO: keep_keys : ['image', 'label', 'length'] [2021/07/01 09:49:23] root INFO: loader : [2021/07/01 09:49:23] root INFO: batch_size_per_card : 512 [2021/07/01 09:49:23] root INFO: drop_last : True [2021/07/01 09:49:23] root INFO: num_workers : 8 [2021/07/01 09:49:23] root INFO: shuffle : True [2021/07/01 09:49:23] root INFO: train with paddle 2.0.2 and device CUDAPlace(0) [2021/07/01 09:49:23] root INFO: Initialize indexs of datasets:['/paddle/data/handwrite/train_data/5660/train.txt'] [2021/07/01 09:49:53] root INFO: Initialize indexs of datasets:['/paddle/data/handwrite/train_data/5660/test.txt'] [2021/07/01 09:50:01] root INFO: load pretrained model from ['/paddle/PaddleOCR/output/rec_chinese_lite_v2.0_myhandwrite_5660_49W/best_accuracy'] [2021/07/01 09:50:01] root INFO: train dataloader has 9738 iters [2021/07/01 09:50:01] root INFO: valid dataloader has 989 iters [2021/07/01 09:50:01] root INFO: During the training process, after the 180000th iteration, an evaluation is run every 30000 iterations [2021/07/01 09:50:01] root INFO: Initialize indexs of datasets:['/paddle/data/handwrite/train_data/5660/train.txt'] [2021/07/01 09:52:07] root INFO: epoch: [1/500], iter: 1, lr: 0.001000, loss: 31.828234, acc: 0.057617, norm_edit_dis: 0.387492, reader_cost: 83.80902 s, batch_cost: 89.17556 s, samples: 1024, ips: 11.48297 [2021/07/01 09:52:12] root INFO: epoch: [1/500], iter: 2, lr: 0.001000, loss: 29.757305, acc: 0.062500, norm_edit_dis: 0.434889, reader_cost: 0.00154 s, batch_cost: 2.27765 s, samples: 512, ips: 224.79344 [2021/07/01 09:52:17] root INFO: epoch: [1/500], iter: 3, lr: 0.001000, loss: 30.260902, acc: 0.059570, norm_edit_dis: 0.415364, reader_cost: 0.00164 s, batch_cost: 2.32385 s, samples: 512, ips: 220.32447 [2021/07/01 09:52:23] root INFO: epoch: [1/500], iter: 4, lr: 0.001000, loss: 30.764500, acc: 0.056641, norm_edit_dis: 0.395838, reader_cost: 0.00296 s, batch_cost: 2.34384 s, samples: 512, ips: 218.44458 [2021/07/01 09:52:29] root INFO: epoch: [1/500], iter: 5, lr: 0.001000, loss: 30.854603, acc: 0.051758, norm_edit_dis: 0.399500, reader_cost: 0.00236 s, batch_cost: 2.25449 s, samples: 512, ips: 227.10237 [2021/07/01 09:52:34] root INFO: epoch: [1/500], iter: 6, lr: 0.001000, loss: 30.913944, acc: 0.054688, norm_edit_dis: 0.403162, reader_cost: 0.00250 s, batch_cost: 2.42188 s, samples: 512, ips: 211.40639 [2021/07/01 09:52:40] root INFO: epoch: [1/500], iter: 7, lr: 0.001000, loss: 30.839222, acc: 0.052734, norm_edit_dis: 0.416499, reader_cost: 0.00201 s, batch_cost: 2.38663 s, samples: 512, ips: 214.52830 [2021/07/01 09:53:26] root INFO: epoch: [1/500], iter: 8, lr: 0.001000, loss: 30.913944, acc: 0.054688, norm_edit_dis: 0.425039, reader_cost: 40.30284 s, batch_cost: 42.81566 s, samples: 512, ips: 11.95824 [2021/07/01 09:53:32] root INFO: epoch: [1/500], iter: 9, lr: 0.001000, loss: 30.839222, acc: 0.053711, norm_edit_dis: 0.423360, reader_cost: 0.00295 s, batch_cost: 2.45239 s, samples: 512, ips: 208.77630 [2021/07/01 09:53:37] root INFO: epoch: [1/500], iter: 10, lr: 0.001000, loss: 30.764500, acc: 0.054688, norm_edit_dis: 0.425039, reader_cost: 0.00686 s, batch_cost: 2.05563 s, samples: 512, ips: 249.07218 [2021/07/01 09:53:43] root INFO: epoch: [1/500], iter: 11, lr: 0.001000, loss: 30.685322, acc: 0.054688, norm_edit_dis: 0.427438, reader_cost: 0.00233 s, batch_cost: 2.47661 s, samples: 512, ips: 206.73415 [2021/07/01 09:53:49] root INFO: epoch: [1/500], iter: 12, lr: 0.001000, loss: 30.764500, acc: 0.054688, norm_edit_dis: 0.429837, reader_cost: 0.00228 s, batch_cost: 2.45082 s, samples: 512, ips: 208.90957 [2021/07/01 09:53:54] root INFO: epoch: [1/500], iter: 13, lr: 0.001000, loss: 30.685322, acc: 0.054688, norm_edit_dis: 0.432363, reader_cost: 0.00220 s, batch_cost: 2.46574 s, samples: 512, ips: 207.64554 [2021/07/01 09:54:00] root INFO: epoch: [1/500], iter: 14, lr: 0.001000, loss: 30.764500, acc: 0.054688, norm_edit_dis: 0.434263, reader_cost: 0.00204 s, batch_cost: 2.39069 s, samples: 512, ips: 214.16372 [2021/07/01 09:54:06] root INFO: epoch: [1/500], iter: 15, lr: 0.001000, loss: 30.685322, acc: 0.054688, norm_edit_dis: 0.434576, reader_cost: 0.00523 s, batch_cost: 2.46833 s, samples: 512, ips: 207.42800 [2021/07/01 09:54:51] root INFO: epoch: [1/500], iter: 16, lr: 0.001000, loss: 30.606144, acc: 0.054688, norm_edit_dis: 0.434889, reader_cost: 39.34539 s, batch_cost: 41.82631 s, samples: 512, ips: 12.24110 [2021/07/01 09:54:56] root INFO: epoch: [1/500], iter: 17, lr: 0.001000, loss: 30.181725, acc: 0.054688, norm_edit_dis: 0.434576, reader_cost: 0.00740 s, batch_cost: 2.15522 s, samples: 512, ips: 237.56267 [2021/07/01 09:55:02] root INFO: epoch: [1/500], iter: 18, lr: 0.001000, loss: 29.757305, acc: 0.054688, norm_edit_dis: 0.434889, reader_cost: 0.00239 s, batch_cost: 2.46737 s, samples: 512, ips: 207.50850 [2021/07/01 09:55:08] root INFO: epoch: [1/500], iter: 19, lr: 0.001000, loss: 29.725346, acc: 0.054688, norm_edit_dis: 0.435060, reader_cost: 0.00256 s, batch_cost: 2.35948 s, samples: 512, ips: 216.99697 [2021/07/01 09:55:14] root INFO: epoch: [1/500], iter: 20, lr: 0.001000, loss: 29.725346, acc: 0.054688, norm_edit_dis: 0.435060, reader_cost: 0.00201 s, batch_cost: 2.36277 s, samples: 512, ips: 216.69524 [2021/07/01 09:55:20] root INFO: epoch: [1/500], iter: 21, lr: 0.001000, loss: 29.619083, acc: 0.055664, norm_edit_dis: 0.435676, reader_cost: 0.00294 s, batch_cost: 2.39487 s, samples: 512, ips: 213.79026 [2021/07/01 09:55:26] root INFO: epoch: [1/500], iter: 22, lr: 0.001000, loss: 29.619083, acc: 0.054688, norm_edit_dis: 0.436227, reader_cost: 0.00218 s, batch_cost: 2.38980 s, samples: 512, ips: 214.24356 [2021/07/01 09:55:31] root INFO: epoch: [1/500], iter: 23, lr: 0.001000, loss: 29.502144, acc: 0.054688, norm_edit_dis: 0.437132, reader_cost: 0.00234 s, batch_cost: 2.31372 s, samples: 512, ips: 221.28896 [2021/07/01 09:56:06] root INFO: epoch: [1/500], iter: 24, lr: 0.001000, loss: 29.442257, acc: 0.055664, norm_edit_dis: 0.438853, reader_cost: 29.38514 s, batch_cost: 31.80103 s, samples: 512, ips: 16.10011 [2021/07/01 09:56:12] root INFO: epoch: [1/500], iter: 25, lr: 0.001000, loss: 29.370758, acc: 0.055664, norm_edit_dis: 0.440730, reader_cost: 0.00602 s, batch_cost: 2.26942 s, samples: 512, ips: 225.60853 [2021/07/01 09:56:17] root INFO: epoch: [1/500], iter: 26, lr: 0.001000, loss: 29.370758, acc: 0.057617, norm_edit_dis: 0.445384, reader_cost: 0.00211 s, batch_cost: 2.35480 s, samples: 512, ips: 217.42848 [2021/07/01 09:56:25] root INFO: epoch: [1/500], iter: 27, lr: 0.001000, loss: 29.442257, acc: 0.059570, norm_edit_dis: 0.445384, reader_cost: 0.87715 s, batch_cost: 3.32005 s, samples: 512, ips: 154.21461 [2021/07/01 09:56:30] root INFO: epoch: [1/500], iter: 28, lr: 0.001000, loss: 29.331280, acc: 0.061523, norm_edit_dis: 0.449332, reader_cost: 0.00205 s, batch_cost: 2.37537 s, samples: 512, ips: 215.54524 [2021/07/01 09:56:36] root INFO: epoch: [1/500], iter: 29, lr: 0.001000, loss: 29.199039, acc: 0.062500, norm_edit_dis: 0.449803, reader_cost: 0.00540 s, batch_cost: 2.08182 s, samples: 512, ips: 245.93842 [2021/07/01 09:56:42] root INFO: epoch: [1/500], iter: 30, lr: 0.001000, loss: 29.331280, acc: 0.062500, norm_edit_dis: 0.449332, reader_cost: 0.00625 s, batch_cost: 2.39618 s, samples: 512, ips: 213.67374 [2021/07/01 09:56:47] root INFO: epoch:

[1/500], iter: 31, lr: 0.001000, loss: 29.331280, acc: 0.062500, norm_edit_dis: 0.449332, reader_cost: 0.00205 s, batch_cost: 2.45351 s, samples: 512, ips: 208.68022

[2021/07/01 09:57:27] root INFO: epoch: [1/500], iter: 32, lr: 0.001000, loss: 29.199039, acc: 0.062500, norm_edit_dis: 0.449893, reader_cost: 34.51909 s, batch_cost: 36.95176 s, samples: 512, ips: 13.85590 image

刚才我将num_workers 改成了 24 时间下降了一些 但是还是不稳定 image

youwei2567 commented 3 years ago

能请教下 1、这个“卡住”怎么解决的吗？ 2、为什么每个epoch都要重新读取并initialize indexes of datasets？

wa3926 commented 3 years ago

慢的问题还没解决 可能是cpu或者磁盘问题 每个 epoch 第一次会慢几秒 重新读取可能是开启图像增广需要具体没细看