search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.87k stars 2.9k forks source link

使用自己的数据集训练ppyoloe_r模型，dfl_loss下降到0.8左右不下降，可视化训练后预测结果角度预测有偏差 #8013

Open ly19940318 opened 1 year ago

ly19940318 commented 1 year ago

问题确认 Search before asking

请提出你的问题 Please ask your question

已完成以下工作： 1）使用DOTA数据集验证，训练过程正常，训练结果与提供的参考指标接近 2）可视化自己的数据集转coco格式后的结果，转换结果正常 3）调整ppyoloe_r_crn.yml文件 loss_weight中dfl权重，dfl_loss值从1.0下降到0.8，但仍存在训练到一定程度dfl_loss不下降问题

部分训练过程如下： arning: Unable to use numba in PP-Tracking, please install numba, for example(python3.7): pip install numba==0.56.4 Warning: Unable to use numba in PP-Tracking, please install numba, for example(python3.7): pip install numba==0.56.4 Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly loading annotations into memory... Done (t=0.04s) creating index... index created! [03/28 14:40:33] ppdet.data.source.coco INFO: Load [1342 samples valid, 0 samples invalid] in file E:\lengyan\dataset\1.WorkPieceDetect\20221222-train\train\COCO\train.json. W0328 14:40:33.505690 5000 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.7, Runtime API Version: 11.6 W0328 14:40:33.578658 5000 gpu_resources.cc:91] device: 0, cuDNN Version: 8.5.

[03/29 08:27:39] ppdet.engine INFO: Epoch: [295] [100/335] learning_rate: 0.001000 loss: 2.374365 loss_cls: 0.532761 loss_iou: 0.065791 loss_dfl: 0.732144 eta: 198 days, 21:17:47 batch_cost: 0.4832 data_cost: 0.0002 ips: 8.2788 images/s [03/29 08:28:10] ppdet.engine INFO: Epoch: [295] [150/335] learning_rate: 0.001000 loss: 2.549767 loss_cls: 0.528721 loss_iou: 0.067839 loss_dfl: 0.801359 eta: 198 days, 21:08:37 batch_cost: 0.4834 data_cost: 0.0001 ips: 8.2753 images/s [03/29 08:28:40] ppdet.engine INFO: Epoch: [295] [200/335] learning_rate: 0.001000 loss: 2.665075 loss_cls: 0.534097 loss_iou: 0.073924 loss_dfl: 0.828117 eta: 198 days, 20:57:40 batch_cost: 0.4770 data_cost: 0.0002 ips: 8.3850 images/s [03/29 08:29:11] ppdet.engine INFO: Epoch: [295] [250/335] learning_rate: 0.001000 loss: 2.486815 loss_cls: 0.540085 loss_iou: 0.071036 loss_dfl: 0.771347 eta: 198 days, 20:48:18 batch_cost: 0.4826 data_cost: 0.0003 ips: 8.2884 images/s [03/29 08:29:42] ppdet.engine INFO: Epoch: [295] [300/335] learning_rate: 0.001000 loss: 2.453403 loss_cls: 0.535102 loss_iou: 0.072839 loss_dfl: 0.761515 eta: 198 days, 20:39:00 batch_cost: 0.4828 data_cost: 0.0000 ips: 8.2843 images/s [03/29 08:30:04] ppdet.engine INFO: Epoch: [296] [ 0/335] learning_rate: 0.001000 loss: 2.480722 loss_cls: 0.537357 loss_iou: 0.071170 loss_dfl: 0.767879 eta: 198 days, 20:35:13 batch_cost: 0.4939 data_cost: 0.0130 ips: 8.0996 images/s [03/29 08:30:35] ppdet.engine INFO: Epoch: [296] [ 50/335] learning_rate: 0.001000 loss: 2.448107 loss_cls: 0.537338 loss_iou: 0.074905 loss_dfl: 0.762847 eta: 198 days, 20:24:02 batch_cost: 0.4760 data_cost: 0.0002 ips: 8.4029 images/s [03/29 08:31:05] ppdet.engine INFO: Epoch: [296] [100/335] learning_rate: 0.001000 loss: 2.518185 loss_cls: 0.531329 loss_iou: 0.067972 loss_dfl: 0.774745 eta: 198 days, 20:13:11 batch_cost: 0.4772 data_cost: 0.0002 ips: 8.3814 images/s [03/29 08:31:35] ppdet.engine INFO: Epoch: [296] [150/335] learning_rate: 0.001000 loss: 2.377278 loss_cls: 0.530690 loss_iou: 0.067600 loss_dfl: 0.740517 eta: 198 days, 20:02:21 batch_cost: 0.4772 data_cost: 0.0001 ips: 8.3821 images/s [03/29 08:32:05] ppdet.engine INFO: Epoch: [296] [200/335] learning_rate: 0.001000 loss: 2.396584 loss_cls: 0.521954 loss_iou: 0.069761 loss_dfl: 0.737611 eta: 198 days, 19:51:46 batch_cost: 0.4781 data_cost: 0.0002 ips: 8.3667 images/s [03/29 08:32:35] ppdet.engine INFO: Epoch: [296] [250/335] learning_rate: 0.001000 loss: 2.692223 loss_cls: 0.518255 loss_iou: 0.068380 loss_dfl: 0.835754 eta: 198 days, 19:41:45 batch_cost: 0.4801 data_cost: 0.0001 ips: 8.3315 images/s [03/29 08:33:06] ppdet.engine INFO: Epoch: [296] [300/335] learning_rate: 0.001000 loss: 2.563722 loss_cls: 0.533303 loss_iou: 0.071386 loss_dfl: 0.794264 eta: 198 days, 19:31:21 batch_cost: 0.4787 data_cost: 0.0002 ips: 8.3566 images/s [03/29 08:33:27] ppdet.engine INFO: Epoch: [297] [ 0/335] learning_rate: 0.001000 loss: 2.221320 loss_cls: 0.531899 loss_iou: 0.066285 loss_dfl: 0.691032 eta: 198 days, 19:25:51 batch_cost: 0.4841 data_cost: 0.0124 ips: 8.2632 images/s [03/29 08:33:57] ppdet.engine INFO: Epoch: [297] [ 50/335] learning_rate: 0.001000 loss: 2.433180 loss_cls: 0.548973 loss_iou: 0.071601 loss_dfl: 0.757860 eta: 198 days, 19:15:38 batch_cost: 0.4793 data_cost: 0.0019 ips: 8.3458 images/s [03/29 08:34:28] ppdet.engine INFO: Epoch: [297] [100/335] learning_rate: 0.001000 loss: 2.524431 loss_cls: 0.526782 loss_iou: 0.070487 loss_dfl: 0.782897 eta: 198 days, 19:05:44 batch_cost: 0.4804 data_cost: 0.0001 ips: 8.3263 images/s [03/29 08:34:58] ppdet.engine INFO: Epoch: [297] [150/335] learning_rate: 0.001000 loss: 2.400022 loss_cls: 0.519908 loss_iou: 0.063310 loss_dfl: 0.737883 eta: 198 days, 18:55:15 batch_cost: 0.4782 data_cost: 0.0001 ips: 8.3644 images/s [03/29 08:35:28] ppdet.engine INFO: Epoch: [297] [200/335] learning_rate: 0.001000 loss: 2.509862 loss_cls: 0.533334 loss_iou: 0.065457 loss_dfl: 0.781662 eta: 198 days, 18:44:00 batch_cost: 0.4755 data_cost: 0.0002 ips: 8.4129 images/s [03/29 08:35:58] ppdet.engine INFO: Epoch: [297] [250/335] learning_rate: 0.001000 loss: 2.568960 loss_cls: 0.524078 loss_iou: 0.066242 loss_dfl: 0.802832 eta: 198 days, 18:32:30 batch_cost: 0.4745 data_cost: 0.0002 ips: 8.4296 images/s [03/29 08:36:28] ppdet.engine INFO: Epoch: [297] [300/335] learning_rate: 0.001000 loss: 2.436484 loss_cls: 0.521943 loss_iou: 0.068217 loss_dfl: 0.754521 eta: 198 days, 18:21:52 batch_cost: 0.4776 data_cost: 0.0001 ips: 8.3751 images/s [03/29 08:36:50] ppdet.engine INFO: Epoch: [298] [ 0/335] learning_rate: 0.001000 loss: 2.476430 loss_cls: 0.514753 loss_iou: 0.067778 loss_dfl: 0.763858 eta: 198 days, 18:16:17 batch_cost: 0.4846 data_cost: 0.0107 ips: 8.2550 images/s [03/29 08:37:21] ppdet.engine INFO: Epoch: [298] [ 50/335] learning_rate: 0.001000 loss: 2.358212 loss_cls: 0.514224 loss_iou: 0.067992 loss_dfl: 0.728640 eta: 198 days, 18:06:27 batch_cost: 0.4804 data_cost: 0.0003 ips: 8.3259 images/s [03/29 08:37:51] ppdet.engine INFO: Epoch: [298] [100/335] learning_rate: 0.001000 loss: 2.545218 loss_cls: 0.525487 loss_iou: 0.063564 loss_dfl: 0.800713 eta: 198 days, 17:55:33 batch_cost: 0.4765 data_cost: 0.0002 ips: 8.3937 images/s [03/29 08:38:21] ppdet.engine INFO: Epoch: [298] [150/335] learning_rate: 0.001000 loss: 2.570733 loss_cls: 0.532586 loss_iou: 0.070173 loss_dfl: 0.803253 eta: 198 days, 17:46:14 batch_cost: 0.4822 data_cost: 0.0001 ips: 8.2960 images/s [03/29 08:38:52] ppdet.engine INFO: Epoch: [298] [200/335] learning_rate: 0.001000 loss: 2.455730 loss_cls: 0.527718 loss_iou: 0.067306 loss_dfl: 0.764158 eta: 198 days, 17:39:29 batch_cost: 0.4914 data_cost: 0.0003 ips: 8.1395 images/s [03/29 08:39:24] ppdet.engine INFO: Epoch: [298] [250/335] learning_rate: 0.001000 loss: 2.633313 loss_cls: 0.523204 loss_iou: 0.069297 loss_dfl: 0.822509 eta: 198 days, 17:37:39 batch_cost: 0.5090 data_cost: 0.0003 ips: 7.8580 images/s [03/29 08:39:56] ppdet.engine INFO: Epoch: [298] [300/335] learning_rate: 0.001000 loss: 2.353439 loss_cls: 0.536395 loss_iou: 0.069272 loss_dfl: 0.732165 eta: 198 days, 17:36:03 batch_cost: 0.5099 data_cost: 0.0003 ips: 7.8454 images/s [03/29 08:40:19] ppdet.engine INFO: Epoch: [299] [ 0/335] learning_rate: 0.001000 loss: 2.478277 loss_cls: 0.540609 loss_iou: 0.073788 loss_dfl: 0.767229 eta: 198 days, 17:37:54

wangxinxin08 commented 1 year ago

loss最终的值还是比较大，可以检查下超参数的设置是否合理

ly19940318 commented 1 year ago

loss最终的值还是比较大，可以检查下超参数的设置是否合理

老师您好，目前使用的参数基本为默认参数。 修改内容如下： base_lr 修改为0.002 （使用单显卡，训练batch=4，根据提供的计算方式计算为0.002） 使用的训练集图像数量为350张，共3个类别，样本的角度分布在0或90左右。 测试集预测结果中角度偏差较大的情况基本出现在实际物体角度为0左右的情况。 想请教下使用的数据集是否合理。

wangxinxin08 commented 1 year ago

数据集太少，建议加载DOTA预训练训练

ly19940318 commented 1 year ago

数据集太少，建议加载DOTA预训练训练

老师您好，按照建议预训练模型加载了https://paddledet.bj.bcebos.com/models/ppyoloe_r_crn_l_3x_dota.pdparams

但是仍会出现问题中的现象，其他参数未做修改。

wangxinxin08 commented 1 year ago

检查下标注，并看下训练集和测试集是不是差异太大了，如果没有以上问题，建议实践下https://aistudio.baidu.com/aistudio/projectdetail/5058293 中在脊柱数据集上的训练

ly19940318 commented 1 year ago

检查下标注，并看下训练集和测试集是不是差异太大了，如果没有以上问题，建议实践下https://aistudio.baidu.com/aistudio/projectdetail/5058293 中在脊柱数据集上的训练

老师您好，按照您的建议先在脊柱数据集上训练结果如下： 脊柱训练数据集 按照推荐的配置配置自定义数据集文件内容如下（修改了epoch）：

_BASE_: [
  '../../datasets/workpiece.yml', # 修改数据集
  '../../runtime.yml',
  '_base_/optimizer_3x.yml',
  '_base_/ppyoloe_r_reader.yml',
  '_base_/ppyoloe_r_crn.yml'
]

log_iter: 50
snapshot_epoch: 1
weights: output/ppyoloe_r_crn_l_3x_dota/model_final

pretrain_weights: https://paddledet.bj.bcebos.com/models/ppyoloe_r_crn_s_3x_dota_ms.pdparams # 使用dota预训练模型，可以加快收敛，在小数据集上提升精度
depth_mult: 0.33
width_mult: 0.50

TrainReader:
  batch_size: 8 # 设置batch_size为8，总的batch_size不变

epoch: 360

LearningRate:
  base_lr: 0.008 # 由于总的batch_size不变，不需要修改学习率
  schedulers:
    - !CosineDecay
      max_epochs: 440
    - !LinearWarmup
      start_factor: 0.
      epochs: 5 # 由于数据集数据较少，我们将warmup设置成5个epoch，而不使用1000个iter

训练结果如下： 自己的数据集

dfl_loss还是下降到一定程度停止下降，可视化推理结果后，有部分角度预测有偏差

ly19940318 commented 1 year ago

预测不准确的结果类似如下： 捕获 角度预测正常的结果类似如下： 捕获22

wangxinxin08 commented 1 year ago

根据你的数据集及训练结果思考下吧，你mAP都到100%了，测试集的mAP能到多少？存在过拟合吗？测试集和训练集是不是差异过大？这些问题都和我们开源的算法本身无关了

maomaoyuchengzi commented 1 year ago

作者你好，我也遇到了这个问题，宽高差异大的检测效果比较好，接近方形的就如你预测不准确的那种图，请问你最后通过调整参数或是数据有所改善吗？

HansenLYX0708 commented 11 months ago

同样的问题，我用training数据集去验证也是有角度偏差

Metavers1 commented 1 month ago

预测不准确的结果类似如下： 捕获 角度预测正常的结果类似如下： 捕获22

兄弟 你知道 目前最新版的 paddlepaddle 3.0beta 和最新的develop的paddledection 该如何配 '../../datasets/workpiece.yml', 这种自己的旋转框的数据集吗 是不是其中的!cocodataset 换成!dota