qarepvgg n模型的trt评估掉点（未进行量化） - Githubissues

meituan / YOLOv6

YOLOv6: a single-stage object detection framework dedicated to industrial applications.

GNU General Public License v3.0

5.71k stars 1.03k forks source link

qarepvgg n模型的trt评估掉点（未进行量化） #755

Open shiyuxin1999S opened 1 year ago

shiyuxin1999S commented 1 year ago

Before Asking

[X] I have read the README carefully. 我已经仔细阅读了README上的操作指引。
[X] I want to train my custom dataset, and I have read the tutorials for training your custom data carefully and organize my dataset correctly; (FYI: We recommand you to apply the config files of xx_finetune.py.) 我想训练自定义数据集，我已经仔细阅读了训练自定义数据的教程，以及按照正确的目录结构存放数据集。（FYI: 我们推荐使用xx_finetune.py等配置文件训练自定义数据集。）
[X] I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking

[X] I have searched the YOLOv6 issues and found no similar questions.

Question

首先我clone了最新的代码：git clone https://github.com/meituan/YOLOv6.git 接下来我修改了coco.yaml：val: ../COCO2017/images/val2017、anno_path: ../COCO2017/annotations/instances_val2017.json 接下来我保证了tool/eval.py与deploy/TensorRT/eval_yolo_trt.py的一致：batch_size=1, img_size=640, conf_thres=0.03, iou_thres=0.65, class_num=80, test_load_size=640, letterbox_return_int=False, scale_exact=False, force_no_pad=False 接下来我导出了onnx模型：python ./deploy/ONNX/export_onnx.py 其中配置为：接下来我导出了trt模型：python deploy/TensorRT/onnx_to_trt.py -m models/yolov6n_qarepvgg.onnx -d fp32 其中配置为：接下来我评估了原始的.pt模型：python tools/eval.py 其中配置为：获得的结果为：接下来我评估了trt模型：python deploy/TensorRT/eval_yolo_trt.py --model models/yolov6n_qarepvgg.trt --is_coco 其中配置为：获得的结果为：

其中yolov6n_qarepvgg.pt是我自行在coco数据集上训练的模型，一切操作如上。我想请问，fp32的trt模型为什么会掉点，是我哪里的细节没有注意到吗，希望得到回复，感谢。 yolov6n_qarepvgg.zip

Additional

No response

mtjhl commented 1 year ago

收到，我也在 coco 数据测试一下，测试完会把结果发这里。 @shiyuxin1999S

xingyueye commented 1 year ago

@shiyuxin1999S 应该是评测没有对齐造成的掉点，包括预处理方式，指标计算方式等等。单独模型转换一般很少会带来精度上的损失。

shiyuxin1999S commented 1 year ago

感谢大家，我好像找到原因了，是评估参数not_infer_on_rect的原因。因为onnx和trt模型的导出没有采用动态shape的方法，所以在eval_yolo_trt.py中必须强制填充灰边，使图像为正方形，导致掉点。随后我自行导出了动态shape的onnx和trt模型，仍然掉点，希望大佬们能帮我看看。测试结果如下： Qarepvgg——yolov6n： Pytorch:batch_size=1, 填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.368 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.524 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.397 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.173 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.409 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.538 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.314 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.523 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.575 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.344 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.781 Pytorch:batch_size=32, 填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.368 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.524 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.397 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.173 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.409 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.538 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.314 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.523 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.575 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.344 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.781 Pytorch:batch_size=32, 不填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.371 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.528 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.402 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.174 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.414 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.545 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.317 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.527 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.578 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.336 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.646 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.785 Pytorch：batch_size=1, 不填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.372 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.528 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.402 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.175 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.415 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.546 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.317 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.526 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.578 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.343 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.646 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.785

Trt：fp32, batch_size=1, 填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.367 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.524 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.397 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.173 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.409 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.538 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.314 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.523 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.575 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.344 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.781

Trt：fp32, batch_size=1, 不填充灰边 Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.368 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.524 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.398 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.172 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.410 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.540 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.315 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.521 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.574 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.342 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.644 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.780

Trt：int8, batch_size=1, 填充灰边, 静态shape Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.359 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.516 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.387 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.166 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.400 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.528 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.308 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.515 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.567 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.326 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.638 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.770

Trt：int8, batch_size=1, 填充灰边, 动态shape Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.353 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.514 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.382 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.165 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.386 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.525 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.305 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.509 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.562 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.328 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.623 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.774

Trt：int8, batch_size=1, 不填充灰边, 动态shape Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.354 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.515 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.382 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.163 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.385 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.528 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.305 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.510 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.562 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.328 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.623 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.775

mtjhl commented 1 year ago

INT8 肯定是会有一些掉点的呀。