integrate with OnnxSlim

Kayzwer commented 2 weeks ago

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

This PR is a placeholder without actual code changes. 🚫

📊 Key Changes

There are no code changes in this Pull Request.

🎯 Purpose & Impact

Since there are no changes, this PR might serve a different purpose, such as documentation, discussion, or project management. 📚
Users and developers should not expect any updates to features, performance, or bug fixes from this PR. No impact on current usage or functionality! 🔄

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 22.22222% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 70.57%. Comparing base (a2ecb24) to head (21034ab).

Files	Patch %	Lines
ultralytics/engine/exporter.py	22.22%	7 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #12683 +/- ## ========================================== + Coverage 70.15% 70.57% +0.41% ========================================== Files 122 122 Lines 15636 15645 +9 ========================================== + Hits 10970 11041 +71 + Misses 4666 4604 -62 ``` | [Flag](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12683/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | Coverage Δ | | |---|---|---| | [Benchmarks](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `35.49% <22.22%> (-0.05%)` | :arrow_down: | | [GPU](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `37.27% <0.00%> (-0.03%)` | :arrow_down: | | [Tests](https://app.codecov.io/gh/ultralytics/ultralytics/pull/12683/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics) | `66.72% <22.22%> (+0.45%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ultralytics#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Ilyabasharov commented 2 weeks ago

@Kayzwer Hello, thx for your effort! What do you think, what is better to use - onnx-simplifier or onnx-slim?

Kayzwer commented 2 weeks ago

@Kayzwer Hello, thx for your effort! What do you think, what is better to use - onnx-simplifier or onnx-slim?

try both

Kayzwer commented 2 weeks ago

@Kayzwer Hello, thx for your effort! What do you think, what is better to use - onnx-simplifier or onnx-slim?

also thanks to @inisis for developing the tool

inisis commented 2 weeks ago

@Kayzwer Hello, thx for your effort! What do you think, what is better to use - onnx-simplifier or onnx-slim?

also thanks to @inisis for developing the tool

Thanks for your pr, and onnxslim can have a better performance.

glenn-jocher commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim?

Are the number of layers, operations or file size different?

Kayzwer commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim?

Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

inisis commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim? Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

Thanks a lot, I have tested myself.

initialencounter commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim? Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

Thanks a lot, I have tested myself.

we have tested all the onnx models generated by tests/test_exports.py, and each model is processd by onnxsim and onnxslim and log to a file, you can find it here.

here is what chatgpt generates:

Final Recommendation

Use onnxsim if you prefer a conservative approach that simplifies the model structure without changing the model size. It ensures the model retains its original constants and is less likely to introduce changes that might affect the model's behavior.

Use onnxslim if you aim for more aggressive optimization, reducing the number of operations more significantly and are willing to accept a slight increase in model size. This approach might yield better performance in terms of computational efficiency at runtime.

Kayzwer commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim? Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

Thanks a lot, I have tested myself.

we have tested all the onnx models generated by tests/test_exports.py, and each model is processd by onnxsim and onnxslim and log to a file, you can find it here.

here is what chatgpt generates:
Final Recommendation

Use onnxsim if you prefer a conservative approach that simplifies the model structure without changing the model size. It ensures the model retains its original constants and is less likely to introduce changes that might affect the model's behavior.

Use onnxslim if you aim for more aggressive optimization, reducing the number of operations more significantly and are willing to accept a slight increase in model size. This approach might yield better performance in terms of computational efficiency at runtime.

but it is possible to apply two at the same time right? (onnxsim first, onnxslim second)

inisis commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim? Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

Thanks a lot, I have tested myself.

we have tested all the onnx models generated by tests/test_exports.py, and each model is processd by onnxsim and onnxslim and log to a file, you can find it here.

here is what chatgpt generates:
Final Recommendation

Use onnxsim if you prefer a conservative approach that simplifies the model structure without changing the model size. It ensures the model retains its original constants and is less likely to introduce changes that might affect the model's behavior.

Use onnxslim if you aim for more aggressive optimization, reducing the number of operations more significantly and are willing to accept a slight increase in model size. This approach might yield better performance in terms of computational efficiency at runtime.

the slimmed model is slightly larger because it contains more detailed tensor value info for the intermediate layer.

initialencounter commented 2 weeks ago

@inisis @Kayzwer thanks guys! Do you have benchmarks or info on i.e. YOLOv8n models exported with both onnx-simplifier and onnx-slim? Are the number of layers, operations or file size different?

@glenn-jocher I will do some experiments with yolov8n.

Thanks a lot, I have tested myself.

we have tested all the onnx models generated by tests/test_exports.py, and each model is processd by onnxsim and onnxslim and log to a file, you can find it here. here is what chatgpt generates:
Final Recommendation

Use onnxsim if you prefer a conservative approach that simplifies the model structure without changing the model size. It ensures the model retains its original constants and is less likely to introduce changes that might affect the model's behavior.

Use onnxslim if you aim for more aggressive optimization, reducing the number of operations more significantly and are willing to accept a slight increase in model size. This approach might yield better performance in terms of computational efficiency at runtime.
but it is possible to apply two at the same time right? (onnxsim first, onnxslim second)

I have conducted an experiment, using yolov8n_False_1.onnx which are non dynamic shape, using sim and slim to process it, and convert them to tensorrt, and here are the result. be423877ca62eebf8d788b0f9a288229 51fae447151387388f1e1dd1164a3fe0 98f57a3b90f7bd4b2f92a18174fceccc 87830b47dc16ea496b95e94a1c47ed14

from the result we can see that using slimmed model, it's 1647.92 qps, and simmed it 1635.28, and eigne file size is 18345652 vs 20483532

glenn-jocher commented 2 weeks ago

@initialencounter hey, thanks for adding to the discussion! Regarding applying onnx-simplifier and onnx-slim sequentially, you can definitely try using both tools on a model to maximize optimization. Each tool removes different redundancies which potentially could complement each other -- applying onnx-simplifier first and then onnx-slim might yield an even more optimized model. However, it's important to thoroughly test the final model to ensure that these optimizations don't impede its accuracy or expected behavior! 🧪🚀

initialencounter commented 2 weeks ago

@initialencounter hey, thanks for adding to the discussion! Regarding applying onnx-simplifier and onnx-slim sequentially, you can definitely try using both tools on a model to maximize optimization. Each tool removes different redundancies which potentially could complement each other -- applying onnx-simplifier first and then onnx-slim might yield an even more optimized model. However, it's important to thoroughly test the final model to ensure that these optimizations don't impede its accuracy or expected behavior! 🧪🚀

@glenn-jocher , all my experiments are done seperately with onnxslim and onnxsim alone and I have tested the accuracy of the exported engine, here are the result, using the script from tests/test_cuda.py.

from the result we can see, that the slimmed engine produces the right result

Kayzwer commented 2 weeks ago

@initialencounter hey, thanks for adding to the discussion! Regarding applying onnx-simplifier and onnx-slim sequentially, you can definitely try using both tools on a model to maximize optimization. Each tool removes different redundancies which potentially could complement each other -- applying onnx-simplifier first and then onnx-slim might yield an even more optimized model. However, it's important to thoroughly test the final model to ensure that these optimizations don't impede its accuracy or expected behavior! 🧪🚀

@glenn-jocher , all my experiments are done seperately with onnxslim and onnxsim alone and I have tested the accuracy of the exported engine, here are the result, using the script from tests/test_cuda.py.

from the result we can see, that the slimmed engine produces the right result

so using only onnxslim can get optimal performance?

glenn-jocher commented 2 weeks ago

@Kayzwer thanks for sharing your findings! Yes, based on your experiments, it seems using only onnxslim has provided satisfactory performance and accuracy. If the slimmed model meets your requirements, you might not need additional optimization with onnx-simplifier. However, different models and applications can benefit variably, so a combined approach could be explored if further optimization is necessary. Keep up the great experimentation! 🚀

inisis commented 2 weeks ago

@glenn-jocher Hi, what else do we need to provide, in order to merge onnxslim into ultralytics

glenn-jocher commented 1 week ago

@inisis the best way to validate both speed and accuracy of the exported models is to export both and then validate on COCO (for both speed and mAP):

yolo export model=yolov8n.pt format=onnx
yolo val model=yolov8n.onnx data=coco.yaml

inisis commented 1 week ago

(py3.10) root@ubuntu20:~# yolo val model=slim_yolov8n.onnx data=coco.yaml
WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'.
Ultralytics YOLOv8.2.18 🚀 Python-3.10.14 torch-2.2.1+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24217MiB)
Loading slim_yolov8n.onnx for ONNX Runtime inference...
Forcing batch=1 square inference (1,3,640,640) for non-PyTorch models
val: Scanning /root/yolov5/datasets/coco/labels/val2017.cache... 4952 images, 48 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:00<?, ?it/s]
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95): 100%|██████████| 5000/5000 [01:08<00:00, 72.92it/s]
                   all       5000      36335      0.629      0.476      0.521       0.37
                person       5000      10777      0.751      0.678      0.745      0.515
               bicycle       5000        314      0.693      0.414      0.466      0.269
                   car       5000       1918      0.656      0.527      0.566      0.363
            motorcycle       5000        367       0.71      0.573      0.654      0.412
              airplane       5000        143      0.748      0.769      0.839      0.654
                   bus       5000        283       0.73      0.664      0.739      0.621
                 train       5000        190      0.795      0.774      0.833      0.648
                 truck       5000        414      0.519      0.384       0.45      0.301
                  boat       5000        424      0.561      0.297      0.373      0.209
         traffic light       5000        634       0.64      0.352      0.415      0.213
          fire hydrant       5000        101      0.859      0.693       0.78      0.627
             stop sign       5000         75      0.678      0.627      0.677      0.615
         parking meter       5000         60      0.687      0.511      0.578      0.449
                 bench       5000        411      0.554      0.275      0.297      0.198
                  bird       5000        427      0.662      0.365      0.426      0.281
                   cat       5000        202      0.767      0.832      0.846      0.647
                   dog       5000        218      0.685      0.693      0.727      0.588
                 horse       5000        272      0.688      0.658       0.69      0.521
                 sheep       5000        354      0.616      0.669      0.669      0.466
                   cow       5000        372      0.714      0.604      0.674       0.48
              elephant       5000        252      0.698      0.843      0.815      0.628
                  bear       5000         71      0.815      0.747      0.833      0.672
                 zebra       5000        266      0.805      0.808      0.879       0.66
               giraffe       5000        232      0.856      0.836      0.884      0.686
              backpack       5000        371      0.494      0.164        0.2      0.105
              umbrella       5000        407       0.61      0.521      0.538      0.359
               handbag       5000        540      0.475      0.122      0.161     0.0815
                   tie       5000        252      0.636      0.377      0.428      0.267
              suitcase       5000        299      0.558      0.425      0.488      0.334
               frisbee       5000        115      0.727      0.757      0.763      0.578
                  skis       5000        241      0.631       0.34      0.377      0.194
             snowboard       5000         69      0.536      0.348      0.381      0.267
           sports ball       5000        260      0.702      0.442      0.481      0.331
                  kite       5000        327      0.612      0.526      0.556      0.379
          baseball bat       5000        145      0.555      0.372      0.411      0.215
        baseball glove       5000        148      0.649      0.486      0.516      0.304
            skateboard       5000        179      0.659      0.592      0.645      0.456
             surfboard       5000        267      0.599      0.476        0.5      0.312
         tennis racket       5000        225      0.677      0.596      0.661      0.403
                bottle       5000       1013      0.603      0.382      0.454      0.298
            wine glass       5000        341      0.667      0.328      0.407      0.263
                   cup       5000        895      0.572      0.438      0.486      0.346
                  fork       5000        215      0.596      0.312      0.375      0.257
                 knife       5000        325      0.448       0.16      0.166     0.0965
                 spoon       5000        253      0.438      0.129      0.162     0.0973
                  bowl       5000        623      0.587      0.485      0.526      0.393
                banana       5000        370      0.555      0.319      0.374      0.228
                 apple       5000        236      0.427       0.23      0.221      0.151
              sandwich       5000        177      0.564      0.467      0.475      0.359
                orange       5000        285      0.471      0.421      0.361      0.274
              broccoli       5000        312      0.507      0.359      0.367       0.21
                carrot       5000        365      0.457      0.285      0.307      0.192
               hot dog       5000        125      0.717      0.406      0.489       0.36
                 pizza       5000        284      0.654      0.616      0.658      0.502
                 donut       5000        328       0.61      0.491      0.516      0.412
                  cake       5000        310       0.56      0.406       0.45        0.3
                 chair       5000       1771      0.577      0.344      0.404      0.259
                 couch       5000        261       0.61      0.567      0.588      0.429
          potted plant       5000        342      0.507      0.374      0.377      0.223
                   bed       5000        163      0.555      0.558        0.6      0.443
          dining table       5000        695      0.524       0.43      0.428      0.287
                toilet       5000        179       0.73      0.725       0.78      0.645
                    tv       5000        288      0.738      0.628      0.724      0.551
                laptop       5000        231       0.69      0.662      0.699      0.578
                 mouse       5000        106      0.662      0.647      0.704      0.521
                remote       5000        283      0.427      0.212      0.283      0.165
              keyboard       5000        153      0.592      0.569       0.65       0.49
            cell phone       5000        262      0.542       0.37      0.405      0.275
             microwave       5000         55      0.661      0.564      0.624      0.499
                  oven       5000        143      0.642      0.497       0.54       0.36
               toaster       5000          9      0.593      0.222      0.422      0.304
                  sink       5000        225      0.578      0.449      0.504      0.328
          refrigerator       5000        126      0.684      0.595      0.659      0.506
                  book       5000       1129      0.457      0.108      0.191     0.0945
                 clock       5000        267      0.727       0.61      0.672      0.459
                  vase       5000        274      0.574      0.474      0.471      0.331
              scissors       5000         36       0.74      0.333      0.342      0.277
            teddy bear       5000        190       0.64      0.574      0.605      0.414
            hair drier       5000         11          1          0    0.00606    0.00425
            toothbrush       5000         57      0.434      0.211      0.218      0.137
Speed: 0.3ms preprocess, 6.4ms inference, 0.0ms loss, 1.6ms postprocess per image
Saving runs/detect/val/predictions.json...

Evaluating pycocotools mAP using runs/detect/val/predictions.json and /root/yolov5/datasets/coco/annotations/instances_val2017.json...
loading annotations into memory...
Done (t=0.43s)
creating index...
index created!
Loading and preparing results...
DONE (t=5.13s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=55.12s).
Accumulating evaluation results...
DONE (t=13.29s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.525
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.405
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.187
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.410
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.533
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.320
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.536
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.592
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.657
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.772
Results saved to runs/detect/val
💡 Learn more at https://docs.ultralytics.com/modes/val

(py3.10) root@ubuntu20:~# yolo val model=sim_yolov8n.onnx data=coco.yaml
WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'.
Ultralytics YOLOv8.2.18 🚀 Python-3.10.14 torch-2.2.1+cu121 CUDA:0 (NVIDIA GeForce RTX 4090, 24217MiB)
Loading sim_yolov8n.onnx for ONNX Runtime inference...
Forcing batch=1 square inference (1,3,640,640) for non-PyTorch models
val: Scanning /root/yolov5/datasets/coco/labels/val2017.cache... 4952 images, 48 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:00<?, ?it/s]
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95): 100%|██████████| 5000/5000 [01:11<00:00, 70.10it/s]
                   all       5000      36335      0.628      0.477      0.521       0.37
                person       5000      10777      0.749      0.679      0.745      0.515
               bicycle       5000        314      0.692      0.414      0.466      0.269
                   car       5000       1918      0.656      0.527      0.566      0.363
            motorcycle       5000        367       0.71      0.575      0.654      0.412
              airplane       5000        143      0.748      0.769      0.839      0.654
                   bus       5000        283      0.729      0.664      0.739      0.621
                 train       5000        190       0.79      0.774      0.833      0.648
                 truck       5000        414      0.518      0.384       0.45      0.301
                  boat       5000        424      0.561      0.298      0.373      0.209
         traffic light       5000        634      0.637      0.352      0.415      0.213
          fire hydrant       5000        101      0.859      0.693       0.78      0.627
             stop sign       5000         75      0.678      0.627      0.677      0.615
         parking meter       5000         60      0.687      0.512      0.578      0.449
                 bench       5000        411      0.551      0.275      0.297      0.197
                  bird       5000        427      0.661      0.365      0.426      0.281
                   cat       5000        202      0.766      0.832      0.846      0.647
                   dog       5000        218      0.684      0.693      0.727      0.588
                 horse       5000        272      0.687      0.658       0.69      0.521
                 sheep       5000        354      0.612      0.669      0.669      0.466
                   cow       5000        372      0.713      0.605      0.674       0.48
              elephant       5000        252      0.698      0.844      0.815      0.628
                  bear       5000         71      0.816      0.748      0.833      0.672
                 zebra       5000        266      0.806       0.81      0.879      0.661
               giraffe       5000        232      0.856      0.836      0.884      0.686
              backpack       5000        371      0.489      0.165        0.2      0.105
              umbrella       5000        407      0.609      0.521      0.538      0.359
               handbag       5000        540      0.473      0.123      0.161     0.0815
                   tie       5000        252      0.635      0.377      0.428      0.267
              suitcase       5000        299      0.558      0.425      0.488      0.334
               frisbee       5000        115      0.727      0.757      0.763      0.578
                  skis       5000        241       0.63       0.34      0.377      0.194
             snowboard       5000         69      0.534      0.348      0.381      0.267
           sports ball       5000        260      0.702      0.446      0.481      0.331
                  kite       5000        327      0.613      0.529      0.556      0.379
          baseball bat       5000        145      0.552      0.372      0.411      0.215
        baseball glove       5000        148      0.648      0.486      0.516      0.304
            skateboard       5000        179      0.659      0.592      0.645      0.456
             surfboard       5000        267      0.598      0.476        0.5      0.312
         tennis racket       5000        225      0.676      0.596      0.661      0.403
                bottle       5000       1013      0.603      0.384      0.454      0.298
            wine glass       5000        341      0.666      0.328      0.407      0.263
                   cup       5000        895      0.571      0.439      0.486      0.346
                  fork       5000        215      0.596      0.312      0.375      0.257
                 knife       5000        325      0.446       0.16      0.166     0.0965
                 spoon       5000        253      0.439       0.13      0.162     0.0973
                  bowl       5000        623      0.586      0.486      0.526      0.393
                banana       5000        370      0.552      0.319      0.374      0.228
                 apple       5000        236      0.428      0.231      0.221      0.151
              sandwich       5000        177      0.564      0.468      0.475      0.359
                orange       5000        285       0.47      0.421      0.361      0.274
              broccoli       5000        312      0.506      0.359      0.367       0.21
                carrot       5000        365      0.455      0.285      0.307      0.192
               hot dog       5000        125      0.718      0.407      0.489       0.36
                 pizza       5000        284      0.653      0.616      0.658      0.502
                 donut       5000        328      0.609      0.491      0.516      0.412
                  cake       5000        310      0.559      0.406       0.45        0.3
                 chair       5000       1771      0.576      0.344      0.404      0.259
                 couch       5000        261      0.607      0.567      0.588      0.429
          potted plant       5000        342      0.507      0.374      0.377      0.223
                   bed       5000        163      0.554      0.558        0.6      0.443
          dining table       5000        695      0.522       0.43      0.428      0.287
                toilet       5000        179       0.73      0.725       0.78      0.645
                    tv       5000        288      0.738      0.628      0.724      0.551
                laptop       5000        231      0.689      0.662      0.699      0.578
                 mouse       5000        106      0.663      0.648      0.704      0.521
                remote       5000        283      0.426      0.212      0.282      0.165
              keyboard       5000        153       0.59      0.569       0.65       0.49
            cell phone       5000        262      0.543      0.372      0.405      0.275
             microwave       5000         55      0.661      0.564      0.624      0.499
                  oven       5000        143       0.64      0.497       0.54       0.36
               toaster       5000          9      0.592      0.222      0.422      0.304
                  sink       5000        225      0.577      0.449      0.504      0.327
          refrigerator       5000        126      0.683      0.595      0.659      0.506
                  book       5000       1129      0.454      0.109      0.191     0.0945
                 clock       5000        267      0.726       0.61      0.672      0.459
                  vase       5000        274      0.574      0.476      0.471      0.331
              scissors       5000         36      0.739      0.333      0.342      0.277
            teddy bear       5000        190       0.64      0.574      0.605      0.414
            hair drier       5000         11          1          0    0.00606    0.00425
            toothbrush       5000         57      0.433      0.211      0.218      0.137
Speed: 0.3ms preprocess, 6.9ms inference, 0.0ms loss, 1.6ms postprocess per image
Saving runs/detect/val2/predictions.json...

Evaluating pycocotools mAP using runs/detect/val2/predictions.json and /root/yolov5/datasets/coco/annotations/instances_val2017.json...
loading annotations into memory...
Done (t=0.42s)
creating index...
index created!
Loading and preparing results...
DONE (t=5.11s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=55.43s).
Accumulating evaluation results...
DONE (t=13.47s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.525
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.405
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.187
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.410
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.533
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.320
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.536
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.592
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.657
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.772
Results saved to runs/detect/val2
💡 Learn more at https://docs.ultralytics.com/modes/val

@glenn-jocher Hi, here are the results, the average inference speed for slim model is 6.4ms while accuracy stay the same.

glenn-jocher commented 1 week ago

Hey @inisis, thanks for sharing the results! It looks like the slim model maintains accuracy while offering a slight improvement in inference speed. This could be beneficial for applications requiring faster processing without a compromise on performance. If there's anything specific you need for integrating onnxslim into Ultralytics, such as further testing or documentation updates, just let me know! 😊🚀

inisis commented 1 week ago

Hey @inisis, thanks for sharing the results! It looks like the slim model maintains accuracy while offering a slight improvement in inference speed. This could be beneficial for applications requiring faster processing without a compromise on performance. If there's anything specific you need for integrating onnxslim into Ultralytics, such as further testing or documentation updates, just let me know! 😊🚀

@glenn-jocher Thanks for your quick response, I'm wondering if there are performance tests for Ultralytics on CI, and we can make a pr which utilize onnxslim to do model optimization and see the speed and accuracy results. Onnxslim is a very useful tool and easy to maintain and cross-platform, users don't need to make much effort to get a speedup. It's very honored to be merged into Ultralytics. 😊🚀😊🚀😊🚀

glenn-jocher commented 1 week ago

Hey @inisis, absolutely, we do have CI performance tests set up for Ultralytics. You're welcome to submit a PR with onnxslim integration! We can then review the changes together, especially focusing on the speed and accuracy benchmarks. It sounds like a great addition to enhance model performance with minimal user effort. Looking forward to your contribution! 😊🚀

inisis commented 1 week ago

Hi, @Kayzwer there is a conflict here, can you help fix it, and I want to make onnxslim as default option, because it's faster.

Kayzwer commented 1 week ago

Hi, @Kayzwer there is a conflict here, can you help fix it, and I want to make onnxslim as default option, because it's faster.

just make slim to true in default.yaml file

inisis commented 1 week ago

Hi, @Kayzwer there is a conflict here, can you help fix it, and I want to make onnxslim as default option, because it's faster.

just make slim to true in default.yaml file

@Kayzwer Can you help fix the merge conflict or should I create another pull request, current workflow is a little bit complicated for me. ~_~

Kayzwer commented 1 week ago

Hi, @Kayzwer there is a conflict here, can you help fix it, and I want to make onnxslim as default option, because it's faster.

just make slim to true in default.yaml file

@Kayzwer Can you help fix the merge conflict or should I create another pull request, current workflow is a little bit complicated for me. ~_~

alright, I will fix it later

glenn-jocher commented 1 week ago

Hi @Kayzwer, thanks for offering to help! If you could resolve the merge conflict, that would be great. Setting slim=true in the default.yaml as the default sounds like a good plan to enhance performance. Let me know if you need any more details from my side! 😊🚀

ultralytics / ultralytics