output labels,scores and boxes of yolov8 ONNX model is out of range with deepsparse

[AtikaGul]

Describe the bug I am trying to transform yolov8n.pt model to yolov8n.onnx to be able to run with deepsparse using ubuntu app (command line interface). I followed the following tutorial: https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/yolov8

I was able to create .onnx model with the command mentioned in the above link and can use it in the code but the outputs of this model is out of range.

Expected behavior For a yolov8 model, I would expect the labels in range of the coco names and scores in range of 100 percent but I am getting output values out of range as follows:

boxes=[[[-19.469205856323242, 1.14739990234375, 35.20930099487305, 87.66728210449219], [-10.840580940246582, -3.1305694580078125, 17.951499938964844, 44.45508575439453], [10.753315925598145, 10.916136741638184, 21.993927001953125, 20.769092559814453], [5.268974781036377, 5.36970853805542, 10.824316024780273, 10.229269027709961]]] scores=[[45513.26171875, 34609.890625, 3955.06396484375, 2994.73095703125]] labels=[['8367.0', '954.0', '8134.0', '6329.0']]

The results I got seems like raw and I can't even know what information can I get from labels in this range. I tried to look to my onnx model in depth using netron.app and found that the output format is float32[1,84,8400] however I am not able to understand how this output relates to the actual yolov8 model output.

Environment

1. OS - Ubuntu desktop app
2. Python version [3.10.6]:
3. DeepSparse version [1.4.0]:

Can anyone please have a look at the issue and guide me in this regards. Many Thanks.

[dbogunowicz]

Hey Atika! Thank's for stress testing our new feature. I will make sure that your problem is resolved.

Few questions:

1. DeepSparse version 1.4.0 only has YOLOv8 functionality showcased under the examples folder (this will change in the next release and will mirror the current situation from the nighly (main branch)). Could you confirm whether you are using 1.4.0 (and thus running YOLOv8 from examples) or are you using the nightly version?
2. what is your ultralytics module version?
3. what is your engine version? This is different from the deepsparse package version. To find out what is the engine version please run:

   deepsparse.benchmark {model_name} 

   e.g

deepsparse.benchmark yolov8n.onnx

You should see the "splash screen that looks like this:

2023-02-27 10:52:31 deepsparse.benchmark.benchmark_model INFO     Thread pinning to cores enabled
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: XXXX... COMMUNITY | (3be6be5f) (release) (optimized) (system=avx2, binary=avx2)
2023-02-27 10:52:33 deepsparse.benchmark.benchmark_model INFO     deepsparse.engine.Engine:

Could you tell me what the "XXXX..." is in your case?

[AtikaGul]

Thank you very much for the quick turnaround @dbogunowicz

I will try to answer the questions you have asked in the following section.

1 - I am using the yolov8 under the 'examples' directory in the gitgub repository however it directed me to the shifter content url (mentioned in the question) and I am following completely that code. I am sure that my DeepSparse version is 1.4.0. I am not sure about the nighly /nightly version so I though I should post my python steps (which I followed from the README file of the repository) from terminal here, It says that there is no "yolov8" option so I had to use "yolo":

`Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from deepsparse import Pipeline import deepsparse deepsparse.version '1.4.0' model_path = '/home/ssembeds_linux/development/yolov8/yolov8n.onnx' images = ['/home/ssembeds_linux/development/yolov8/basilica.jpg']

yolo_pipeline = Pipeline.create(task='yolov8',model_path=model_path,) Traceback (most recent call last): File "", line 1, in File "/home/ssembeds_linux/.local/lib/python3.10/site-packages/deepsparse/pipeline.py", line 456, in create pipeline_constructor = Pipeline._get_task_constructor(task) File "/home/ssembeds_linux/.local/lib/python3.10/site-packages/deepsparse/pipeline.py", line 407, in _get_task_constructor SupportedTasks.check_register_task(task, _REGISTERED_PIPELINES.keys()) File "/home/ssembeds_linux/.local/lib/python3.10/site-packages/deepsparse/tasks.py", line 183, in check_register_task raise ValueError( ValueError: Unknown Pipeline task yolov8. Currently supported tasks are ['embedding_extraction', 'text_classification', 'token_classification', 'ner', 'custom', 'transformers_embedding_extraction', 'yolo', 'zero_shot_text_classification', 'yolact', 'qa', 'open_pif_paf', 'glue', 'image_classification', 'haystack', 'sentiment_analysis', 'question_answering', 'information_retrieval_haystack']

yolo_pipeline = Pipeline.create(task='yolo',model_path=model_path,) DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.4.0 COMMUNITY | (3be6be5f) (release) (optimized) (system=avx512_vnni, binary=avx512) pipeline_outputs = yolo_pipeline(images=images)

print(pipeline_outputs) boxes=[[[-19.469205856323242, 1.14739990234375, 35.20930099487305, 87.66728210449219], [-10.840580940246582, -3.1305694580078125, 17.951499938964844, 44.45508575439453], [10.753315925598145, 10.916136741638184, 21.993927001953125, 20.769092559814453], [5.268974781036377, 5.36970853805542, 10.824316024780273, 10.229269027709961]]] scores=[[45513.26171875, 34609.890625, 3955.06396484375, 2994.73095703125]] labels=[['8367.0', '954.0', '8134.0', '6329.0']]

`

2 - The ultralytics version is 8.0.39 `>>> import ultralytics

ultralytics.version '8.0.39' `

3 - output of the command:

deepsparse.benchmark {model_name}

is the following:

`:~$ deepsparse.benchmark /home/ssembeds_linux/development/yolov8/yolov8n.onnx

2023-02-28 09:53:28 deepsparse.benchmark.benchmark_model INFO Thread pinning to cores enabled DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.4.0 COMMUNITY | (3be6be5f) (release) (optimized) (system=avx512_vnni, binary=avx512) 2023-02-28 09:53:29 deepsparse.benchmark.benchmark_model INFO deepsparse.engine.Engine: onnx_file_path: /home/ssembeds_linux/development/yolov8/yolov8n.onnx batch_size: 1 num_cores: 4 num_streams: 1 scheduler: Scheduler.default fraction_of_supported_ops: 1.0 cpu_avx_type: avx512 cpu_vnni: True 2023-02-28 09:53:29 deepsparse.utils.onnx INFO Generating input 'images', type = float32, shape = [1, 3, 640, 640] 2023-02-28 09:53:29 deepsparse.benchmark.benchmark_model INFO Starting 'singlestream' performance measurements for 10 seconds Original Model Path: /home/ssembeds_linux/development/yolov8/yolov8n.onnx Batch Size: 1 Scenario: sync Throughput (items/sec): 23.3350 Latency Mean (ms/batch): 42.8342 Latency Median (ms/batch): 42.6767 Latency Std (ms/batch): 2.6606 Iterations: 234 `

Please let me know if I can provide any further information or do any further tests. Thank you!

[dbogunowicz]

Oh, the issue is that you are using the yolov8 onnx model with the yolo (v5) pipeline! This happens here (citing your stack trace):

yolo_pipeline = Pipeline.create(task='yolo',model_path=model_path,)

YOLOv8 integration is a new feature, so we do not have exhaustive documentation around it. Actually, it makes sense that you took this path :) There is a subtle difference between their outputs (v8 and v5), but not that much so that an error is thrown.

I propose you forget about 1.4 for now and do the following thing:

1. pip install deepsparse-nightly[yolov8] This will install the nightly version (the dev version that has not been officially released yet). Should be usable, and could be replaced soon by our 1.5 release. The yolov8 dependency makes sure that you have the adequate e.g. ultralytics version
2. Now export the onnx file using ultralytics library as you previously did.
3. Finally:

   from deepsparse import Pipeline
   img = "src/deepsparse/yolo/sample_images/basilica.jpg"
   pipeline = Pipeline.create(task="yolov8", model_path="yolov8n.onnx")
   out = pipeline(images=img)

   out
   >> YOLOOutput(boxes=[[[262.8804931640625, 481.8933389186859, 518.8655090332031, 611.0700168609619], [730.7918701171875, 407.02944922447205, 769.2503356933594, 494.0161454677582], [334.36141204833984, 394.2888569831848, 449.39154052734375, 587.4708032608032], [310.1635437011719, 394.24361991882324, 363.6779479980469, 437.62432956695557], [57.7701530456543, 408.99205350875854, 98.22649383544922, 499.4660611152649], [18.602170944213867, 338.83033561706543, 42.04681205749512, 387.2004451751709], [466.4468078613281, 384.4201521873474, 530.0118713378906, 408.28860807418823], [95.12386322021484, 381.8725242614746, 134.87340545654297, 462.99033212661743], [712.8079833984375, 397.64378786087036, 739.1504516601562, 472.75402879714966], [547.5377197265625, 385.3713719844818, 589.486083984375, 411.43730759620667], [471.7704162597656, 464.77686309814453, 574.1974182128906, 537.0913686752319], [794.9514770507812, 383.08750581741333, 858.3568725585938, 490.1339030265808], [648.9877624511719, 397.7334747314453, 691.8333435058594, 498.1498260498047], [690.0027008056641, 402.71085381507874, 720.0617523193359, 470.3082902431488], [201.1237335205078, 385.2265467643738, 227.0836944580078, 441.2429919242859]]], scores=[[0.8504291772842407, 0.8041958808898926, 0.7190808653831482, 0.7052393555641174, 0.6656487584114075, 0.6578878164291382, 0.6294032335281372, 0.6176968812942505, 0.5987032055854797, 0.5770964622497559, 0.563414990901947, 0.49874573945999146, 0.48700156807899475, 0.46442314982414246, 0.4520561993122101]], labels=[['3.0', '0.0', '0.0', '2.0', '0.0', '9.0', '2.0', '0.0', '0.0', '2.0', '3.0', '0.0', '0.0', '0.0', '0.0']])

[AtikaGul]

Perfectly done ! Thank you so much @dbogunowicz, You have done a great job bringing me through all the steps and resolved the issue I was having. I really love the work done by deepsparse library and I am getting excellent FPS now on my windows machine running Ubuntu app.

So, instead of using :

pip install deepsparse[yolov8]

We use:

pip install deepsparse-nightly[yolov8]

And the issue resolved ! Thanks again.

[dbogunowicz]

Great to hear that you succeeded! Nothing makes us happier than high FPS numbers! 🚀 If you remain interested in YOLOv8, we are working hard on making the inference faster and our UI better. Expect more documentation, YOLOv8 tasks (e.g. instance segmentation), and sparse models to come out very very soon!

Cheers

[AtikaGul]

Thank you!, Yes very happy with the results to be honest. I am going to use yolov8 a lot in my near future projects and will have a close look at this repository and documentation to release in near time. Excited to know that you are working new tasks and models. You are doing awesome work Thanks!

[AtikaGul]

Hello @dbogunowicz I have been doing testing with my onnx models and is going great as for as the working of framework is concerned however it seems like the accuracy go down for the same model when we convert it to onnx model.

I have a little challenging dataset of a room from overhead images which yolov8n.pt or yolov8m.pt does not work however yolov8x.pt works well with detection. When I convert this yolov8x.pt to .onnx, it is not detecting the same objects from the same images. Do you have any recommendation in this regard or if you can connect me to an existing issue ( I was not able to find one relevant ). or shall I open a new issue for this ?

Thank you very much for the amazing work you are doing and for the support.

[dbogunowicz]

Let's continue talking here. Could you give me a code snippet to reproduce?

[AtikaGul]

Thank you very much for the quick response as always. I was preparing some results to show you here.

I trained a model for our custom dataset which contain 4 classes and got the model in .pt format. then used the following command to convert it to .onnx as normal.

yolo task=detect mode=export model=best_yol_s.pt format=onnx opset=13

Then I got two codes, one is running onnx model using deepsparse and the other is running pt model using ultralytics.

Following is the code and results for pt model: `import os import time import sys from ultralytics import YOLO import cv2 from supervision.tools.detections import Detections HOME = os.getcwd() model = YOLO(f'{HOME}/yolov8_options/best_yol_s.pt') label_names = model.names min_conf = 0.35

img = cv2.imread('couch.png') dres = model(img)

detections = Detections( xyxy=dres[0].boxes.xyxy.cpu().numpy(), confidence=dres[0].boxes.conf.cpu().numpy(), class_id=dres[0].boxes.cls.cpu().numpy().astype(int) ) for det in detections: conf = float(det[1]) cls = int(det[2]) box = det[0] if conf > min_conf: cl_id = '{}: {}'.format(label_names[cls], str(round(conf, 2))) bbox = [int(bx) for bx in box] img = cv2.rectangle(img, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (200, 50, 50), 2) img = cv2.putText(img, cl_id, (bbox[0], bbox[1] + 10), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (200, 50, 50), 2, cv2.LINE_AA) cv2.imshow('pt model output', img) key = cv2.waitKey(0)

cv2.destroyAllWindows() `

While the code and result for onnx model is following:

`import time from deepsparse import Pipeline as dsp import cv2 import sys min_conf = 0.35

pipeline = dsp.create( task='yolov8', model_path='yolov8_onnx_options/best_yol_s.onnx', )

img = cv2.imread('couch.png') pipeline_outputs =pipeline(images = [img], iou_thres=0.6, conf_thres=0.001) print(pipeline_outputs) boxes = pipeline_outputs.boxes[0] scores = pipeline_outputs.scores[0] labels = pipeline_outputs.labels[0]

for j in range(len(labels)): lb = int(float(labels[j])) if scores[j] > min_conf: bbox = [int(bx) for bx in boxes[j]] img = cv2.rectangle(img, (bbox[0], bbox[1]), (bbox[2],bbox[3]), (200, 50, 50), 2)

    cl_id = '{}: {}'.format(lb, str(round(scores[j], 2)))
    frame = cv2.putText(img, cl_id, (bbox[0], bbox[1] + 10), cv2.FONT_HERSHEY_SIMPLEX,
                        0.8, (200, 50, 50), 2, cv2.LINE_AA)

cv2.imshow('onnx model output',img) key = cv2.waitKey(0) cv2.destroyAllWindows() `

There is a difference in results of both these models while I am assuming the same response from onnx model as I have from the pt model.

I also have tried using general yolov8n(x)(m).pt models by converting them to onnx format as above but the results has a significant difference. Can you please have a look at what I am doing wrong here ? Thank you!

[dbogunowicz]

Hey @AtikaGul. Having some things on my plate, but I am aware of this issue, will try to resolve it ASAP.

[AtikaGul]

Thanks for getting back @dbogunowicz ! Okay I can understand. you have been doing a great job. please feel free to ask if you need anything for testing and I will do it right away. Thanks again

[thusinh1969]

nightly build works :) Thanks.

[dbogunowicz]

Glad to hear! Will close this issue for now.