paddlex训练收敛比paddledetetion慢

lidingke commented 2 years ago

环境 paddlepaddle2.2,paddlex2.1.0,paddledeteciton,release/2.4. ubuntu18.04 3.9 CUDA10.2 在同样的参数下，在自己的数据集上paddlex的MAP收敛比paddledetectionMap慢十几个点。因此在fruit数据集上也测试了一下,发现同样收敛慢很多，同样是在200轮，paddledetection的bestMAP为81.9%，paddleX的bestMAP为73.1%，整个训练过程中paddleX的map比paddledetection一直差10个点左右。都使用的是ppyolotiny ################### paddleX的参数。 TRAIN_FILE_LIST = "./dataset/fruit/train.txt" VAL_FILE_LIST = "./dataset/fruit/val.txt" LABEL_LIST = "./dataset/fruit/label_list.txt" CKPT_DIR = "output" DATA_DIR = "./dataset/fruit/" w,h,c = (512,512,3) epochs = 200 batch_size=12 learning_rate_init = 0.005 backbone = "MobileNetV3" train_transforms = T.Compose([ T.RandomDistort(), T.RandomHorizontalFlip(), T.RandomVerticalFlip(), T.BatchRandomResize( target_sizes=[384, 416, 448, 480, 512, 544, 576, 608, 640, 672], interp='RANDOM'), T.Normalize( mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) eval_transforms = T.Compose([ T.Resize( w, interp='CUBIC'), T.Normalize( mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) train_dataset = pdx.datasets.VOCDetection( data_dir=DATA_DIR, file_list=TRAIN_FILE_LIST, label_list=LABEL_LIST, transforms=train_transforms, shuffle=True, ) eval_dataset = pdx.datasets.VOCDetection( data_dir=DATA_DIR, file_list=VAL_FILE_LIST, label_list=LABEL_LIST, transforms=eval_transforms, )

num_classes = len(train_dataset.labels)
print("num_classes:",num_classes)
model = pdx.det.PPYOLOTiny(num_classes=num_classes, backbone=backbone,)#in_c = c)

model.train(
    num_epochs=epochs,
    train_dataset=train_dataset,
    train_batch_size=batch_size,
    eval_dataset=eval_dataset,
    learning_rate=learning_rate_init,
    lr_decay_epochs=[210, 240],
    warmup_steps= 4000,
    warmup_start_lr=0.0,
    save_dir=CKPT_DIR,
    use_vdl=False,
    save_interval_epochs=1,
    use_ema=True,
    )

paddledetection的参数主要修改的是ppyolo_tiny_reader.yml，其他的参数只是修改了路径。 worker_num: 4 TrainReader: inputs_def: num_max_boxes: 100 sample_transforms:

Decode: {}
- Mixup: {alpha: 1.5, beta: 1.5}
RandomDistort: {}
- RandomExpand: {fill_value: [123.675, 116.28, 103.53]}
RandomCrop: {}
RandomFlip: {} batch_transforms:
BatchRandomResize: target_size: [384, 416, 448, 480, 512, 544, 576, 608, 640, 672] random_size: True random_interp: True keep_ratio: False
NormalizeBox: {}
PadBox: {num_max_boxes: 100}
BboxXYXY2XYWH: {}
NormalizeImage: {mean: [0.485, 0.456, 0.406], std: [0.229, 0.224, 0.225], is_scale: True}
Permute: {}
Gt2YoloTarget: anchor_masks: [[6, 7, 8], [3, 4, 5], [0, 1, 2]] anchors: [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45], [59, 119], [116, 90], [156, 198], [373, 326]] downsample_ratios: [32, 16, 8] batch_size: 12 shuffle: true drop_last: true
mixup_epoch: 500

use_shared_memory: true

EvalReader: sample_transforms:

Decode: {}
Resize: {target_size: [512, 512], keep_ratio: False, interp: 2}
NormalizeImage: {mean: [0.485, 0.456, 0.406], std: [0.229, 0.224, 0.225], is_scale: True}
Permute: {} batch_size: 8 collate_batch: false

TestReader: inputs_def: image_shape: [3, 512, 512] sample_transforms:

Decode: {}
Resize: {target_size: [512, 512], keep_ratio: False, interp: 2}
NormalizeImage: {mean: [0.485, 0.456, 0.406], std: [0.229, 0.224, 0.225], is_scale: True}
Permute: {} batch_size: 8 collate_batch: false

因为已经调整并测试过好几个参数，希望能给出原因和解决方案。

lidingke commented 2 years ago

paddledetection第200轮log [05/20 16:07:48] ppdet.engine INFO: Epoch: [198] [ 0/20] learning_rate: 0.004950 loss_xy: 0.925429 loss_wh: 0.577886 loss_iou: 2.276000 loss_obj: 1.623148 loss_cls: 0.264911 loss: 5.437475 eta: 0:43:25 batch_cost: 0.2469 data_cost: 0.0355 ips: 48.5978 images/s [05/20 16:07:54] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_tiny_650e_fruit_voc [05/20 16:07:54] ppdet.engine INFO: Eval iter: 0 [05/20 16:07:55] ppdet.metrics.metrics INFO: Accumulating evaluatation results... [05/20 16:07:55] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 80.67% [05/20 16:07:55] ppdet.engine INFO: Total sample number: 60, averge FPS: 56.95465412807577 [05/20 16:07:55] ppdet.engine INFO: Best test bbox ap is 0.819. [05/20 16:07:55] ppdet.engine INFO: Epoch: [199] [ 0/20] learning_rate: 0.004975 loss_xy: 1.048889 loss_wh: 0.630377 loss_iou: 2.418828 loss_obj: 1.709188 loss_cls: 0.347068 loss: 6.363063 eta: 0:43:17 batch_cost: 0.2492 data_cost: 0.0225 ips: 48.1513 images/s [05/20 16:08:02] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_tiny_650e_fruit_voc [05/20 16:08:02] ppdet.engine INFO: Eval iter: 0 [05/20 16:08:03] ppdet.metrics.metrics INFO: Accumulating evaluatation results... [05/20 16:08:03] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 80.84% [05/20 16:08:03] ppdet.engine INFO: Total sample number: 60, averge FPS: 61.32637552152518 [05/20 16:08:03] ppdet.engine INFO: Best test bbox ap is 0.819. [05/20 16:08:04] ppdet.engine INFO: Epoch: [200] [ 0/20] learning_rate: 0.005000 loss_xy: 0.922640 loss_wh: 0.552581 loss_iou: 2.146456 loss_obj: 1.347513 loss_cls: 0.310493 loss: 5.451310 eta: 0:43:11 batch_cost: 0.2717 data_cost: 0.0615 ips: 44.1721 images/s [05/20 16:08:11] ppdet.utils.checkpoint INFO: Save checkpoint: output/ppyolo_tiny_650e_fruit_voc [05/20 16:08:11] ppdet.engine INFO: Eval iter: 0 [05/20 16:08:12] ppdet.metrics.metrics INFO: Accumulating evaluatation results... [05/20 16:08:12] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 80.94% [05/20 16:08:12] ppdet.engine INFO: Total sample number: 60, averge FPS: 63.68057362007589 [05/20 16:08:12] ppdet.engine INFO: Best test bbox ap is 0.819.

paddleX第200轮log 2022-05-20 16:32:01 [INFO] [TRAIN] Epoch=197/200, Step=10/20, loss_xy=1.400873, loss_wh=1.037215, loss_iou=3.959072, loss_obj=2.129635, loss_cls=0.180567, loss=8.707363, lr=0.004911, time_each_step=1.01s, eta=0:1:19 2022-05-20 16:32:10 [INFO] [TRAIN] Epoch=197/200, Step=20/20, loss_xy=1.150931, loss_wh=0.739432, loss_iou=2.731311, loss_obj=2.173386, loss_cls=0.434669, loss=7.229728, lr=0.004924, time_each_step=0.93s, eta=0:1:5 2022-05-20 16:32:10 [INFO] [TRAIN] Epoch 197 finished, loss_xy=1.4046829, loss_wh=1.0663426, loss_iou=3.7790456, loss_obj=3.3011336, loss_cls=0.71820694, loss=10.269411 . 2022-05-20 16:32:10 [WARNING] Detector only supports single card evaluation with batch_size=1 during evaluation, so batch_size is forcibly set to 1. 2022-05-20 16:32:11 [INFO] Start to evaluate(total_samples=60, total_steps=60)... 2022-05-20 16:32:14 [INFO] Accumulating evaluatation results... 2022-05-20 16:32:14 [INFO] [EVAL] Finished, Epoch=197, bbox_map=71.601200 . 2022-05-20 16:32:14 [INFO] Current evaluated best model on eval_dataset is epoch_158, bbox_map=72.91953750331731 2022-05-20 16:32:14 [INFO] Model saved in output/epoch_197. 2022-05-20 16:32:24 [INFO] [TRAIN] Epoch=198/200, Step=10/20, loss_xy=1.347791, loss_wh=0.668129, loss_iou=2.814017, loss_obj=1.906480, loss_cls=0.613221, loss=7.349639, lr=0.004936, time_each_step=1.0s, eta=0:0:56 2022-05-20 16:32:32 [INFO] [TRAIN] Epoch=198/200, Step=20/20, loss_xy=1.309611, loss_wh=0.912018, loss_iou=3.603856, loss_obj=1.618526, loss_cls=0.947147, loss=8.391159, lr=0.004949, time_each_step=0.86s, eta=0:0:40 2022-05-20 16:32:32 [INFO] [TRAIN] Epoch 198 finished, loss_xy=1.4363143, loss_wh=1.1317133, loss_iou=3.8347104, loss_obj=3.3799024, loss_cls=0.769878, loss=10.552519 . 2022-05-20 16:32:33 [WARNING] Detector only supports single card evaluation with batch_size=1 during evaluation, so batch_size is forcibly set to 1. 2022-05-20 16:32:33 [INFO] Start to evaluate(total_samples=60, total_steps=60)... 2022-05-20 16:32:36 [INFO] Accumulating evaluatation results... 2022-05-20 16:32:36 [INFO] [EVAL] Finished, Epoch=198, bbox_map=72.375413 . 2022-05-20 16:32:36 [INFO] Current evaluated best model on eval_dataset is epoch_158, bbox_map=72.91953750331731 2022-05-20 16:32:36 [INFO] Model saved in output/epoch_198. 2022-05-20 16:32:47 [INFO] [TRAIN] Epoch=199/200, Step=10/20, loss_xy=0.894317, loss_wh=0.895505, loss_iou=2.685461, loss_obj=3.676730, loss_cls=0.295666, loss=8.447680, lr=0.004961, time_each_step=1.1s, eta=0:0:36 2022-05-20 16:32:54 [INFO] [TRAIN] Epoch=199/200, Step=20/20, loss_xy=1.498911, loss_wh=1.136554, loss_iou=3.823327, loss_obj=4.496077, loss_cls=1.336598, loss=12.291467, lr=0.004974, time_each_step=0.7s, eta=0:0:17 2022-05-20 16:32:54 [INFO] [TRAIN] Epoch 199 finished, loss_xy=1.4114132, loss_wh=1.0542355, loss_iou=3.7807918, loss_obj=3.209612, loss_cls=0.8352804, loss=10.291332 . 2022-05-20 16:32:54 [WARNING] Detector only supports single card evaluation with batch_size=1 during evaluation, so batch_size is forcibly set to 1. 2022-05-20 16:32:54 [INFO] Start to evaluate(total_samples=60, total_steps=60)... 2022-05-20 16:32:57 [INFO] Accumulating evaluatation results... 2022-05-20 16:32:57 [INFO] [EVAL] Finished, Epoch=199, bbox_map=73.121409 . 2022-05-20 16:32:57 [INFO] Model saved in output/best_model. 2022-05-20 16:32:57 [INFO] Current evaluated best model on eval_dataset is epoch_199, bbox_map=73.12140905659392 2022-05-20 16:32:58 [INFO] Model saved in output/epoch_199. 2022-05-20 16:33:07 [INFO] [TRAIN] Epoch=200/200, Step=10/20, loss_xy=1.923847, loss_wh=1.442866, loss_iou=5.283563, loss_obj=4.556926, loss_cls=1.159819, loss=14.367021, lr=0.004986, time_each_step=0.89s, eta=0:0:8 2022-05-20 16:33:13 [INFO] [TRAIN] Epoch=200/200, Step=20/20, loss_xy=1.434782, loss_wh=0.801290, loss_iou=3.286917, loss_obj=2.546371, loss_cls=0.619517, loss=8.688877, lr=0.004999, time_each_step=0.69s, eta=0:0:0 2022-05-20 16:33:13 [INFO] [TRAIN] Epoch 200 finished, loss_xy=1.3495948, loss_wh=1.0949217, loss_iou=3.8701038, loss_obj=3.204811, loss_cls=0.78117764, loss=10.300609 . 2022-05-20 16:33:14 [WARNING] Detector only supports single card evaluation with batch_size=1 during evaluation, so batch_size is forcibly set to 1. 2022-05-20 16:33:14 [INFO] Start to evaluate(total_samples=60, total_steps=60)... 2022-05-20 16:33:17 [INFO] Accumulating evaluatation results... 2022-05-20 16:33:17 [INFO] [EVAL] Finished, Epoch=200, bbox_map=72.491484 . 2022-05-20 16:33:17 [INFO] Current evaluated best model on eval_dataset is epoch_199, bbox_map=73.12140905659392 2022-05-20 16:33:17 [INFO] Model saved in output/epoch_200.

FlyingQianMM commented 2 years ago

PaddleDetection使用的完整yml文件发一下吧。收敛不一致可能是lr、bs、pretrained_weights等不一致，需要对比下完整的参数

lidingke commented 2 years ago

只修改了ppyolo_tiny_reader.yml，参数也发了，学习率batchsize和pretrained都是一样的。您完全可以自己试一试，因为在公开的fruit数据集上测试的也是这个结论。

lidingke commented 2 years ago

PaddleDetection使用的完整yml文件发一下吧。收敛不一致可能是lr、bs、pretrained_weights等不一致，需要对比下完整的参数

只修改了ppyolo_tiny_reader.yml，其他的参数保持和这里不变https://github.com/PaddlePaddle/PaddleDetection/blob/release/2.4/configs/ppyolo/ppyolo_tiny_650e_coco.yml

PaddlePaddle / PaddleX

paddlex训练收敛比paddledetetion慢 #1507

- Mixup: {alpha: 1.5, beta: 1.5}

- RandomExpand: {fill_value: [123.675, 116.28, 103.53]}

mixup_epoch: 500