自训练RKYOLOv5模型在RK3568平台部署过程中出现的目标检测结果标注框不显示问题

[MrMzl]

环境

• 【FastDeploy版本】： fastdeploy-develop版本
• 【系统平台】: Linux firefly 4.19.232 / Debian 10
• 【硬件】： firefly AIO-3568J
• 【编译语言】：C++ & Python(3.7）

问题日志及出现问题的操作流程

c++部署正常运行后，能够在命令窗口打印推理结果，但是保存的结果图上无目标检测框，另外通过窗口输出的结果进行分析，输出结果个数与真实推理图中的目标数量不一致。命令窗口打印出来的推理结果中目标框很多，但实际所用的推理图中目标只有10个左右.

• 命令窗口输出内容 firefly@firefly:~/FastDeploy_RK3568_FLYCV/FastDeploy-develop/examples/vision/detection/rkyolo/cpp/build/install$ ./infer_rkyolo model/yolov5s_3568_rgb_car_RK356X_i8.rknn images/rgb_car/0003.jpg [INFO] fastdeploy/vision/common/processors/base.cc(97)::EnableFlyCV Will change to use image processing library ProcLib::FLYCV [INFO] fastdeploy/runtime/backends/rknpu2/rknpu2_backend.cc(57)::GetSDKAndDeviceVersion rknn_api/rknnrt version: 1.4.0 (a10f100eb@2022-09-09T09:07:14), driver version: 0.7.2 index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922, pass_through=0 index=0, name=output, n_dims=4, dims=[1, 18, 80, 80], n_elems=115200, size=115200, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922, pass_through=0 index=1, name=272, n_dims=4, dims=[1, 18, 40, 40], n_elems=28800, size=28800, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922, pass_through=0 index=2, name=274, n_dims=4, dims=[1, 18, 20, 20], n_elems=7200, size=7200, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922, pass_through=0 [INFO] fastdeploy/runtime/runtime.cc(334)::CreateRKNPU2Backend Runtime initialized with Backend::RKNPU2 in Device::RKNPU. [WARNING] fastdeploy/runtime/backends/rknpu2/rknpu2_backend.cc(326)::Infer The input tensor type != model's inputs type.The input_type need INT8,but inputs[0].type is UINT8 [FastDeploy] RKYOLOV5 in RKNN duration = 1.70816s. DetectionResult: [xmin, ymin, xmax, ymax, score, label_id] 85.847931,585.966614, 118.152061, 614.033386, 0.999997, 1 110.775131,585.224854, 141.224869, 614.775146, 0.999997, 1 124.547310,581.789978, 151.452682, 618.210022, 0.999997, 1 136.547302,582.614502, 163.452682, 617.385498, 0.999997, 1 173.392273,576.441895, 198.607697, 623.558044, 0.999997, 1 149.392273,581.789978, 174.607712, 618.210022, 0.999997, 1 71.911331,560.494507, 108.088669, 615.505432, 0.999997, 1 99.674927,565.380981, 128.325073, 610.618896, 0.999997, 1 159.674927,570.614502, 188.325073, 605.385498, 0.999997, 1 184.547302,569.789978, 211.452682, 606.210022, 0.999997, 1 197.392288,571.684082, 222.607727, 604.315857, 0.999997, 1 210.209854,571.684082, 233.790131, 604.315857, 0.999997, 1 221.392288,572.464050, 246.607727, 603.535950, 0.999997, 1 233.392288,572.464050, 258.607727, 603.535950, 0.999997, 1 245.392288,572.464050, 270.607727, 603.535950, 0.999997, 1 256.547302,571.684082, 283.452667, 604.315857, 0.999997, 1 267.674927,569.789978, 296.325073, 606.210022, 0.999997, 1 276.893341,567.201843, 311.106659, 608.798035, 0.999997, 1 302.775116,571.684082, 333.224854, 604.315857, 0.999997, 1 314.775116,573.966614, 345.224854, 602.033386, 0.999997, 1 327.674927,575.392700, 356.325073, 600.607239, 0.999997, 1 342.209839,577.388733, 365.790131, 598.611328, 0.999997, 1 352.547302,574.689209, 379.452667, 601.310730, 0.999997, 1 364.547302,577.388733, 391.452698, 598.611328, 0.999997, 1 378.209839,573.966614, 401.790161, 602.033386, 0.999997, 1 390.209839,571.684082, 413.790100, 604.315857, 0.999997, 1 403.000031,576.742493, 424.999969, 599.257507, 0.999997, 1 415.762787,577.388733, 436.237183, 598.611328, 0.999997, 1 427.000031,574.689209, 448.999969, 601.310730, 0.999997, 1 438.209839,573.224854, 461.790100, 602.775146, 0.999997, 1 449.392273,571.684082, 474.607666, 604.315857, 0.999997, 1 458.775146,568.083679, 489.224884, 607.916260, 0.999997, 1 472.547272,562.506409, 499.452667, 613.493591, 0.999997, 1 482.775146,570.614502, 513.224854, 605.385498, 0.999997, 1 496.547272,568.083679, 523.452637, 607.916260, 0.999997, 1 163.000031,580.083679, 184.999969, 619.916260, 0.999997, 1 504.893311,573.224854, 539.106628, 602.775146, 0.999997, 1 513.865112,568.946411, 554.134888, 607.053589, 0.999997, 1 538.901917,568.083679, 577.098083, 607.916260, 0.999997, 1 552.893311,571.684082, 587.106628, 604.315857, 0.999997, 1 566.775146,576.742493, 597.224854, 599.257507, 0.999997, 1 576.893311,579.212769, 611.106628, 596.787109, 0.999997, 1 587.911316,576.742493, 624.088623, 599.257507, 0.999997, 1 601.847900,574.689209, 634.152039, 601.310730, 0.999997, 1 246.209854,588.742493, 269.790131, 611.257507, 0.999997, 1 615.674927,571.684082, 644.325073, 604.315857, 0.999997, 1 627.674927,573.224854, 656.325073, 602.775146, 0.999997, 1 639.674927,574.689209, 668.325073, 601.310730, 0.999997, 1 650.775146,574.689209, 681.224854, 601.310730, 0.999997, 1 664.547241,573.966614, 691.452637, 602.033386, 0.999997, 1 677.392273,571.684082, 702.607666, 604.315857, 0.999997, 1 689.392273,571.684082, 714.607666, 604.315857, 0.999997, 1 701.392273,573.224854, 726.607666, 602.775146, 0.999997, 1 713.392273,573.224854, 738.607666, 602.775146, 0.999997, 1 723.674927,571.684082, 752.325073, 604.315857, 0.999997, 1 736.547241,571.684082, 763.452637, 604.315857, 0.999997, 1 747.674927,569.789978, 776.325073, 606.210022, 0.999997, 1 757.847900,572.464050, 790.152100, 603.535950, 0.999997, 1 768.893372,576.742493, 803.106689, 599.257507, 0.999997, 1 794.775146,573.966614, 825.224854, 602.033386, 0.999997, 1 807.674927,570.614502, 836.325073, 605.385498, 0.999997, 1 258.209839,585.966614, 281.790131, 614.033386, 0.999997, 1 304.547302,593.708557, 331.452667, 606.291382, 0.999997, 1 269.392273,588.742493, 294.607727, 611.257507, 0.999997, 1 821.392273,568.946411, 846.607727, 607.053589, 0.999997, 1 222.209854,587.392700, 245.790131, 612.607239, 0.999997, 1 832.547302,568.946411, 859.452759, 607.053589, 0.999997, 1 842.775146,568.946411, 873.224854, 607.053589, 0.999997, 1 855.674927,570.614502, 884.325073, 605.385498, 0.999997, 1 869.392273,570.614502, 894.607727, 605.385498, 0.999997, 1 882.209839,573.224854, 905.790161, 602.775146, 0.999997, 1 895.762756,568.083679, 916.237122, 607.916260, 0.999997, 1 908.256042,570.614502, 927.743835, 605.385498, 0.999997, 1 919.000000,570.614502, 940.999939, 605.385498, 0.999997, 1 930.209839,570.614502, 953.790161, 605.385498, 0.999997, 1 280.547302,591.782593, 307.452667, 608.217285, 0.999997, 1 292.547302,594.189209, 319.452667, 605.810669, 0.999997, 1 0.000000,578.300964, 10.152063, 621.698975, 0.999997, 1 0.000000,574.506409, 23.106661, 625.493591, 0.999997, 1 0.000000,580.946411, 37.098072, 619.053589, 0.999997, 1 30.209862,579.201843, 53.790138, 620.798035, 0.999997, 1 36.893337,582.614502, 71.106659, 617.385498, 0.999997, 1 45.865112,589.388733, 86.134880, 610.611328, 0.999997, 1 62.775135,631.684082, 93.224869, 664.315857, 0.999997, 1 50.775135,633.966614, 81.224869, 662.033386, 0.999997, 1 24.893337,635.392700, 59.106663, 660.607239, 0.999997, 1 10.901925,632.464050, 49.098072, 663.535950, 0.999997, 1 0.000000,624.441895, 37.098072, 671.558044, 0.999997, 1 0.000000,623.483704, 10.152063, 672.516235, 0.999997, 1 925.847900,616.083679, 958.151978, 655.916260, 0.999997, 1 901.847900,620.464050, 934.151978, 651.535950, 0.999997, 1 887.911377,622.689209, 924.088684, 649.310730, 0.999997, 1 863.911377,630.189209, 900.088684, 641.810669, 0.999997, 1 851.911377,635.616943, 888.088684, 636.383057, 0.999997, 1 827.911377,629.208801, 864.088684, 642.791199, 0.999997, 1 817.847900,623.392700, 850.151978, 648.607239, 0.999997, 1 806.775146,621.224854, 837.224854, 650.775146, 0.999997, 1 796.547302,617.789978, 823.452759, 654.210022, 0.999997, 1 784.547302,621.224854, 811.452759, 650.775146, 0.999997, 1 771.674927,623.392700, 800.325073, 648.607239, 0.999997, 1 760.547241,624.742493, 787.452637, 647.257507, 0.999997, 1 748.547241,624.742493, 775.452637, 647.257507, 0.999997, 1 737.392273,629.208801, 762.607666, 642.791199, 0.999997, 1 725.392273,630.189209, 750.607666, 641.810669, 0.999997, 1 711.674927,628.333374, 740.325073, 643.666626, 0.999997, 1 701.392273,622.689209, 726.607666, 649.310730, 0.999997, 1 689.392273,621.224854, 714.607666, 650.775146, 0.999997, 1 676.547241,618.614502, 703.452637, 653.385498, 0.999997, 1 661.847900,617.789978, 694.152039, 654.210022, 0.999997, 1 649.847900,615.201843, 682.152039, 656.798035, 0.999997, 1 637.847900,616.946411, 670.152039, 655.053589, 0.999997, 1 608.800842,620.464050, 651.199036, 651.535950, 0.999997, 1 586.901917,623.392700, 625.098083, 648.607239, 0.999997, 1 578.775146,620.464050, 609.224854, 651.535950, 0.999997, 1 567.674927,620.464050, 596.325073, 651.535950, 0.999997, 1 553.847900,625.388733, 586.152039, 646.611328, 0.999997, 1 542.775146,624.742493, 573.224854, 647.257507, 0.999997, 1 531.674927,622.689209, 560.325073, 649.310730, 0.999997, 1 518.775146,621.224854, 549.224854, 650.775146, 0.999997, 1 506.775146,624.742493, 537.224854, 647.257507, 0.999997, 1 492.893311,614.300964, 527.106628, 657.698975, 0.999997, 1 481.847931,617.789978, 514.152039, 654.210022, 0.999997, 1 471.674927,616.083679, 500.325073, 655.916260, 0.999997, 1 460.547272,616.946411, 487.452667, 655.053589, 0.999997, 1 447.674927,616.946411, 476.325073, 655.053589, 0.999997, 1 435.674927,616.946411, 464.325073, 655.053589, 0.999997, 1 423.674927,616.083679, 452.325073, 655.916260, 0.999997, 1 410.775146,614.300964, 441.224884, 657.698975, 0.999997, 1 398.775146,619.684082, 429.224884, 652.315857, 0.999997, 1 384.893311,615.201843, 419.106628, 656.798035, 0.999997, 1 359.911316,613.380981, 396.088654, 658.618896, 0.999997, 1 338.775116,620.464050, 369.224854, 651.535950, 0.999997, 1 326.775116,616.083679, 357.224854, 655.916260, 0.999997, 1 315.674927,615.201843, 344.325073, 656.798035, 0.999997, 1 303.674927,615.201843, 332.325073, 656.798035, 0.999997, 1 292.547302,615.201843, 319.452667, 656.798035, 0.999997, 1 280.547302,614.300964, 307.452667, 657.698975, 0.999997, 1 268.547302,616.946411, 295.452667, 655.053589, 0.999997, 1 256.547302,626.623901, 283.452667, 645.376099, 0.999997, 1 243.674927,633.851257, 272.325073, 638.148804, 0.999997, 1 230.775116,635.693298, 261.224854, 636.306580, 0.999997, 1 219.674927,635.575562, 248.325073, 636.424438, 0.999997, 1 210.209854,627.782593, 233.790131, 644.217285, 0.999997, 1 197.392288,621.966614, 222.607727, 650.033386, 0.999997, 1 184.547302,625.388733, 211.452682, 646.611328, 0.999997, 1 171.674927,621.966614, 200.325073, 650.033386, 0.999997, 1 158.775131,614.300964, 189.224869, 657.698975, 0.999997, 1 140.800888,600.463867, 183.199097, 671.536072, 0.999997, 1 128.800888,572.305847, 171.199097, 699.694153, 0.999997, 1 109.847931,612.441895, 142.152054, 659.558044, 0.999997, 1 100.547310,617.789978, 127.452690, 654.210022, 0.999997, 1 89.392281,617.789978, 114.607712, 654.210022, 0.999997, 1 76.547310,619.684082, 103.452690, 652.315857, 0.999997, 1 52.547310,618.614502, 79.452690, 653.385498, 0.999997, 1 39.674919,624.077148, 68.325073, 647.922791, 0.999997, 1 26.775131,633.851257, 57.224865, 638.148804, 0.999997, 1 12.893338,624.077148, 47.106663, 647.922791, 0.999997, 1 0.000000,619.684082, 21.224867, 652.315857, 0.999997, 1 915.674927,606.614502, 944.325073, 641.385498, 0.999997, 1 903.674927,607.684082, 932.325073, 640.315857, 0.999997, 1 891.674927,605.789978, 920.325073, 642.210022, 0.999997, 1 877.847900,598.506409, 910.151978, 649.493591, 0.999997, 1 863.911377,586.495483, 900.088684, 661.504456, 0.999997, 1 851.911377,557.679565, 888.088684, 690.320435, 0.999997, 1 830.775146,588.463867, 861.224854, 659.536072, 0.999997, 1 759.674927,606.614502, 788.325073, 641.385498, 0.999997, 1 748.547241,610.689209, 775.452637, 637.310730, 0.999997, 1 736.547241,615.212769, 763.452637, 632.787109, 0.999997, 1 721.847900,600.441895, 754.152039, 647.558044, 0.999997, 1 698.775146,602.300964, 729.224854, 645.698975, 0.999997, 1 687.674927,604.946411, 716.325073, 643.053589, 0.999997, 1 679.000000,612.742493, 700.999939, 635.257507, 0.999997, 1 664.547241,622.980286, 691.452637, 625.019775, 0.999997, 1 651.674927,623.486389, 680.325073, 624.513489, 0.999997, 1 641.392273,614.623901, 666.607666, 633.376099, 0.999997, 1 626.775146,607.684082, 657.224854, 640.315857, 0.999997, 1 602.775146,606.614502, 633.224854, 641.385498, 0.999997, 1 592.547241,604.946411, 619.452637, 643.053589, 0.999997, 1 581.392273,607.684082, 606.607666, 640.315857, 0.999997, 1 557.392273,604.946411, 582.607666, 643.053589, 0.999997, 1 545.392273,604.946411, 570.607666, 643.053589, 0.999997, 1 533.392273,605.789978, 558.607666, 642.210022, 0.999997, 1 521.392273,618.499146, 546.607666, 629.500916, 0.999997, 1 508.547272,621.945618, 535.452637, 626.054321, 0.999997, 1 400.547272,607.684082, 427.452667, 640.315857, 0.999997, 1 388.547272,605.789978, 415.452667, 642.210022, 0.999997, 1 377.392273,597.510010, 402.607727, 650.489929, 0.999997, 1 353.392273,604.946411, 378.607727, 643.053589, 0.999997, 1 341.392273,605.789978, 366.607727, 642.210022, 0.999997, 1 256.547302,616.333374, 283.452667, 631.666626, 0.999997, 1 244.547302,615.782593, 271.452667, 632.217285, 0.999997, 1 232.547302,616.864990, 259.452667, 631.135071, 0.999997, 1 222.209854,584.474121, 245.790131, 663.525940, 0.999997, 1 209.392288,604.946411, 234.607727, 643.053589, 0.999997, 1 198.209854,618.189209, 221.790131, 629.810669, 0.999997, 1 185.392273,615.782593, 210.607697, 632.217285, 0.999997, 1 173.392273,615.782593, 198.607697, 632.217285, 0.999997, 1 161.392273,613.388733, 186.607712, 634.611328, 0.999997, 1 149.392273,605.789978, 174.607712, 642.210022, 0.999997, 1 136.547302,606.614502, 163.452682, 641.385498, 0.999997, 1

Visualized result saved in ./vis_result.jpg

[Zheng-Bicheng]

框会不会有重合的情况？

[MrMzl]

有的，我在另一个场景数据训练rkyolov5s模型进行部署验证后会出现大量重合的检测框，这一点可以在生成的结果图上得到验证。换到现在这个场景下生成的结果图上不显示标注框，应该也是有重合的。

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2023年03月15日 12:41 | | 收件人 | @.> | | 抄送至 | @.>@.> | | 主题 | Re: [PaddlePaddle/FastDeploy] 自训练RKYOLOv5模型在RK3568平台部署过程中出现的目标检测结果标注框不显示问题 (Issue #1611) |

框会不会有重合的情况？

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[Zheng-Bicheng]

• 先用训练完的模型，在PC上跑一下FP32的推理看看有没有问题
• 再转换FP16的模型，在RKNN上看一下有没有问题
• 最后再看一下int8的模型有没有问题

[Zheng-Bicheng]

能解决问题吗？没猜错应该是模型的问题？

[MrMzl]

嗯，应该就是模型的问题，我这边用rk官方的YOLOv5代码训练自己的数据和模型，训练验证都没有问题，但就是导出onnx模型带上参数--rknpu指定硬件后，导出的onnx模型就推理失败了，不带反倒推理正确。

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2023年03月25日 21:15 | | 收件人 | @.> | | 抄送至 | @.>、State @.> | | 主题 | Re: [PaddlePaddle/FastDeploy] 自训练RKYOLOv5模型在RK3568平台部署过程中出现的目标检测结果标注框不显示问题 (Issue #1611) |

能解决问题吗？没猜错应该是模型的问题？

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>