search

TexasInstruments / edgeai-modeloptimization

This repository has been moved. The new location is in https://github.com/TexasInstruments/edgeai-tensorlab
https://github.com/TexasInstruments/edgeai
Other
0 stars 0 forks source link

Yolov8 model conversion #2

Open Mugutech62 opened 6 months ago

Mugutech62 commented 6 months ago

Hello @mathmanu I have seen the some of the models including Yolov8 in your model surgery repo, I have my yolov8 model, How can I use this repo to make my model compatible with edge device TDA4VM, Can you guide me to achieve this Thanks in advance

mathmanu commented 6 months ago

Yes - the model surgery in this support YOLOV8 from this repository: https://github.com/open-mmlab/mmyolo In fact several models from https://github.com/open-mmlab/mmyolo are supported.

Mugutech62 commented 6 months ago

Hi @mathmanu I have trained yolov8 from ultralytics repository, Is there any way to port them on TDA4VM Thanks

mathmanu commented 6 months ago

Do you mean the original YOLOV8 without surgery - I think some of the layers such as SiLU activation may not currently work in TIDL. However, if you apply surgery and then train, it might work - but we have not checked in that repository. But we have checked in the mmyolo repository. Note: The optimization YOLOV8 detection head will be supported in the upcoming TIDL 9.2 release only in April 2024 beginning (next month). But even without that, those layers can run on ARM.

Mugutech62 commented 6 months ago

Do you mean the original YOLOV8 without surgery - I think some of the layers such as SiLU activation may not currently work in TIDL. However, if you apply surgery and then train, it might work - but we have not checked in that repository. But we have checked in the mmyolo repository. Note: The optimization YOLOV8 detection head will be supported in the upcoming TIDL 9.2 release only in April 2024 beginning (next month). But even without that, those layers can run on ARM.

Thanks Manu Mathew! How can i compile yolov8 then, to run unsupported layers on arm and supported layers on dsp? Is there any way to achieve this? thanks in advance

mathmanu commented 6 months ago

You can use the onnxruntime with TIDL support: https://github.com/TexasInstruments/edgeai-tidl-tools

Mugutech62 commented 6 months ago

You can use the onnxruntime with TIDL support: https://github.com/TexasInstruments/edgeai-tidl-tools

I am having onnx model of my yolov8, kindly can you please explain the steps and procedure 🙏

Mugutech62 commented 6 months ago

You can use the onnxruntime with TIDL support: https://github.com/TexasInstruments/edgeai-tidl-tools

Hello @mathmanu As you suggested to try compilation with edge ai tidl tools repository, I got some exception during compilation, Screenshot from 2024-03-12 16-24-03 Can you help us to solve this issue Attachements: runtimes_visualization

Mugutech62 commented 6 months ago

You can use the onnxruntime with TIDL support: https://github.com/TexasInstruments/edgeai-tidl-tools

Hello @mathmanu did you have any idea about this?

Mugutech62 commented 6 months ago

You can use the onnxruntime with TIDL support: https://github.com/TexasInstruments/edgeai-tidl-tools

Hello @mathmanu As you suggested to try compilation with edge ai tidl tools repository, I got some exception during compilation, Screenshot from 2024-03-12 16-24-03 Can you help us to solve this issue Attachements: runtimes_visualization

Hello @mathmanu Is there any way to help us in regarding solving this?

mathmanu commented 6 months ago

One error that I can see in the picture that you posted is unsupported opset version - which TIDL version are you using?

In the picture of he model that you posted, I can see several Sigmoid and Mult layers - which indicates the presence of SiLU layers in the onnx model. That means model surgery has not been applied.

Mugutech62 commented 6 months ago

One error that I can see in the picture that you posted is unsupported opset version - which TIDL version are you using?

In the picture of he model that you posted, I can see several Sigmoid and Mult layers - which indicates the presence of SiLU layers in the onnx model. That means model surgery has not been applied.

Hello @mathmanu Thanks for the reply, I am currently using Sdk 9.1 latest version, can I use onnx modifier to replace sigmoid node graphically with relu? Also is it possible to guide as how to load yolov8 using model surgery Thanks

mathmanu commented 6 months ago
  1. As discussed in the other thread, you can use that function to replace the layer names to avoid any potential buffer overflow: https://github.com/TexasInstruments/edgeai-benchmark/issues/14
  2. For the model surgery, you can use one of these surgery tools in this repository on the PyTorch model (before training).

Please read the documentation this repository.

Mugutech62 commented 6 months ago
  1. As discussed in the other thread, you can use that function to replace the layer names to avoid any potential buffer overflow: https://github.com/TexasInstruments/edgeai-benchmark/issues/14
  2. For the model surgery, you can use one of these surgery tools in this repository on the PyTorch model (before training).

Please read the documentation this repository. Hello @mathmanu I could see from yolov8 documentation that, we can replace any act fn, so I am going to replace with relu, what about the slice operator? I got some issues related offset version, thank you

mathmanu commented 6 months ago

If a certain opset_version is not supported in TIDL, you can use the opset_version argument during onnx export to a lower version - for example opset_version=11

Mugutech62 commented 6 months ago

If a certain opset_version is not supported in TIDL, you can use the opset_version argument during onnx export to a lower version - for example opset_version=11

Superve @mathmanu thanks you, we will try it and let you know

Mugutech62 commented 6 months ago

Hello @mathmanu We replaced Silu activation layers with Relu and we have exported the model to required opset version 11 onnx format. Then we tried to compile onnx file with edge tidl tools, the compilation occurs upto some extent and crashes after generating some files in artifacts folder. I have ziped the entire directory for your reference with some output logs too. We specified Maxpool layer in deny list since we had kernal shape with 5x5. Output logs: `mugu@mugu-ASUS:~/edgeai-tidl-tools$ python3 compile_model.py /home/mugu/edgeai-tidl-tools/best_names.onnx /home/mugu/edgeai-tidl-tools/test_data /home/mugu/edgeai-tidl-tools/model-artifacts/ Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider'] tidl_tools_path = /home/mugu/edgeai-tidl-tools/tidl_tools artifacts_folder = /home/mugu/edgeai-tidl-tools/model-artifacts/tidl_output tidl_tensor_bits = 8 debug_level = 3 num_tidl_subgraphs = 16 tidl_denylist = MaxPool
tidl_denylist_layer_name = tidl_denylist_layer_type = tidl_allowlist_layer_name = model_type =
tidl_calibration_accuracy_level = 7 tidl_calibration_options:num_frames_calibration = 21 tidl_calibration_options:bias_calibration_iterations = 5 mixed_precision_factor = -1.000000 model_group_id = 0 power_of_2_quantization = 2 ONNX QDQ Enabled = 0 enable_high_resolution_optimization = 0 pre_batchnorm_fold = 1 add_data_convert_ops = 0 output_feature_16bit_names_list =
m_params_16bit_names_list =
reserved_compile_constraints_flag = 1601 ti_internal_reserved_1 =

** WARNING : Network not identified as Object Detection network : (1) Ignore if network is not Object Detection network (2) If network is Object Detection network, please specify "model_type":"OD" as part of OSRT compilation options**

Supported TIDL layer type --- Conv -- /model.0/conv/Conv Supported TIDL layer type --- Relu -- /model.0/act/Relu Supported TIDL layer type --- Conv -- /model.1/conv/Conv Supported TIDL layer type --- Relu -- /model.1/act/Relu Supported TIDL layer type --- Conv -- /model.2/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.2/cv1/act/Relu Supported TIDL layer type --- Split -- /model.2/Split Supported TIDL layer type --- Conv -- /model.2/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.2/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.2/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.2/m.0/cv2/act/Relu Supported TIDL layer type --- Add -- /model.2/m.0/Add Supported TIDL layer type --- Concat -- /model.2/Concat Supported TIDL layer type --- Conv -- /model.2/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.2/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.3/conv/Conv Supported TIDL layer type --- Relu -- /model.3/act/Relu Supported TIDL layer type --- Conv -- /model.4/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.4/cv1/act/Relu Supported TIDL layer type --- Split -- /model.4/Split Supported TIDL layer type --- Conv -- /model.4/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.4/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.4/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.4/m.0/cv2/act/Relu Supported TIDL layer type --- Add -- /model.4/m.0/Add Supported TIDL layer type --- Conv -- /model.4/m.1/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.4/m.1/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.4/m.1/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.4/m.1/cv2/act/Relu Supported TIDL layer type --- Add -- /model.4/m.1/Add Supported TIDL layer type --- Concat -- /model.4/Concat Supported TIDL layer type --- Conv -- /model.4/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.4/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.5/conv/Conv Supported TIDL layer type --- Relu -- /model.5/act/Relu Supported TIDL layer type --- Conv -- /model.6/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.6/cv1/act/Relu Supported TIDL layer type --- Split -- /model.6/Split Supported TIDL layer type --- Conv -- /model.6/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.6/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.6/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.6/m.0/cv2/act/Relu Supported TIDL layer type --- Add -- /model.6/m.0/Add Supported TIDL layer type --- Conv -- /model.6/m.1/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.6/m.1/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.6/m.1/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.6/m.1/cv2/act/Relu Supported TIDL layer type --- Add -- /model.6/m.1/Add Supported TIDL layer type --- Concat -- /model.6/Concat Supported TIDL layer type --- Conv -- /model.6/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.6/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.7/conv/Conv Supported TIDL layer type --- Relu -- /model.7/act/Relu Supported TIDL layer type --- Conv -- /model.8/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.8/cv1/act/Relu Supported TIDL layer type --- Split -- /model.8/Split Supported TIDL layer type --- Conv -- /model.8/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.8/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.8/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.8/m.0/cv2/act/Relu Supported TIDL layer type --- Add -- /model.8/m.0/Add Supported TIDL layer type --- Concat -- /model.8/Concat Supported TIDL layer type --- Conv -- /model.8/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.8/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.9/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.9/cv1/act/Relu Op type 'MaxPool' added to unsupported nodes as specified in deny list Op type 'MaxPool' added to unsupported nodes as specified in deny list Op type 'MaxPool' added to unsupported nodes as specified in deny list Supported TIDL layer type --- Concat -- /model.9/Concat Supported TIDL layer type --- Conv -- /model.9/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.9/cv2/act/Relu Supported TIDL layer type --- Resize -- /model.10/Resize Supported TIDL layer type --- Concat -- /model.11/Concat Supported TIDL layer type --- Conv -- /model.12/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.12/cv1/act/Relu Supported TIDL layer type --- Split -- /model.12/Split Supported TIDL layer type --- Conv -- /model.12/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.12/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.12/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.12/m.0/cv2/act/Relu Supported TIDL layer type --- Concat -- /model.12/Concat Supported TIDL layer type --- Conv -- /model.12/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.12/cv2/act/Relu Supported TIDL layer type --- Resize -- /model.13/Resize Supported TIDL layer type --- Concat -- /model.14/Concat Supported TIDL layer type --- Conv -- /model.15/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.15/cv1/act/Relu Supported TIDL layer type --- Split -- /model.15/Split Supported TIDL layer type --- Conv -- /model.15/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.15/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.15/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.15/m.0/cv2/act/Relu Supported TIDL layer type --- Concat -- /model.15/Concat Supported TIDL layer type --- Conv -- /model.15/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.15/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.16/conv/Conv Supported TIDL layer type --- Relu -- /model.16/act/Relu Supported TIDL layer type --- Concat -- /model.17/Concat Supported TIDL layer type --- Conv -- /model.18/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.18/cv1/act/Relu Supported TIDL layer type --- Split -- /model.18/Split Supported TIDL layer type --- Conv -- /model.18/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.18/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.18/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.18/m.0/cv2/act/Relu Supported TIDL layer type --- Concat -- /model.18/Concat Supported TIDL layer type --- Conv -- /model.18/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.18/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.19/conv/Conv Supported TIDL layer type --- Relu -- /model.19/act/Relu Supported TIDL layer type --- Concat -- /model.20/Concat Supported TIDL layer type --- Conv -- /model.21/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.21/cv1/act/Relu Supported TIDL layer type --- Split -- /model.21/Split Supported TIDL layer type --- Conv -- /model.21/m.0/cv1/conv/Conv Supported TIDL layer type --- Relu -- /model.21/m.0/cv1/act/Relu Supported TIDL layer type --- Conv -- /model.21/m.0/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.21/m.0/cv2/act/Relu Supported TIDL layer type --- Concat -- /model.21/Concat Supported TIDL layer type --- Conv -- /model.21/cv2/conv/Conv Supported TIDL layer type --- Relu -- /model.21/cv2/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.2/cv3.2.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.2/cv3.2.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.2/cv3.2.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.2/cv3.2.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.2/cv3.2.2/Conv Supported TIDL layer type --- Conv -- /model.22/cv2.2/cv2.2.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.2/cv2.2.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.2/cv2.2.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.2/cv2.2.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.2/cv2.2.2/Conv Supported TIDL layer type --- Concat -- /model.22/Concat_2 Supported TIDL layer type --- Reshape -- /model.22/Reshape_2 Supported TIDL layer type --- Conv -- /model.22/cv3.1/cv3.1.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.1/cv3.1.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.1/cv3.1.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.1/cv3.1.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.1/cv3.1.2/Conv Supported TIDL layer type --- Conv -- /model.22/cv2.1/cv2.1.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.1/cv2.1.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.1/cv2.1.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.1/cv2.1.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.1/cv2.1.2/Conv Supported TIDL layer type --- Concat -- /model.22/Concat_1 Supported TIDL layer type --- Reshape -- /model.22/Reshape_1 Supported TIDL layer type --- Conv -- /model.22/cv3.0/cv3.0.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.0/cv3.0.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.0/cv3.0.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv3.0/cv3.0.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv3.0/cv3.0.2/Conv Supported TIDL layer type --- Conv -- /model.22/cv2.0/cv2.0.0/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.0/cv2.0.0/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.0/cv2.0.1/conv/Conv Supported TIDL layer type --- Relu -- /model.22/cv2.0/cv2.0.1/act/Relu Supported TIDL layer type --- Conv -- /model.22/cv2.0/cv2.0.2/Conv Supported TIDL layer type --- Concat -- /model.22/Concat Supported TIDL layer type --- Reshape -- /model.22/Reshape Supported TIDL layer type --- Concat -- /model.22/Concat_3 Supported TIDL layer type --- Split -- /model.22/Split Supported TIDL layer type --- Sigmoid -- /model.22/Sigmoid Supported TIDL layer type --- Reshape -- /model.22/dfl/Reshape Supported TIDL layer type --- Transpose -- /model.22/dfl/Transpose Supported TIDL layer type --- Softmax -- /model.22/dfl/Softmax Supported TIDL layer type --- Transpose -- /model.22/dfl/Transpose_1 Supported TIDL layer type --- Conv -- /model.22/dfl/conv/Conv Supported TIDL layer type --- Reshape -- /model.22/dfl/Reshape_1 Supported TIDL layer type --- Slice -- /model.22/Slice_1 Supported TIDL layer type --- Add -- /model.22/Add_1 Supported TIDL layer type --- Slice -- /model.22/Slice Supported TIDL layer type --- Sub -- /model.22/Sub Supported TIDL layer type --- Sub -- /model.22/Sub_1 Supported TIDL layer type --- Add -- /model.22/Add_2 Supported TIDL layer type --- Div -- /model.22/Div_1 Supported TIDL layer type --- Concat -- /model.22/Concat_4 Supported TIDL layer type --- Mul -- /model.22/Mul_2 Supported TIDL layer type --- Concat -- /model.22/Concat_5

Preliminary subgraphs created = 2 Final number of subgraphs created are : 2, - Offloaded Nodes - 174, Total Nodes - 177 Node in deny list...delegated to ARM --- layer type - MaxPool, Node name - /model.9/m/MaxPool
Node in deny list...delegated to ARM --- layer type - MaxPool, Node name - /model.9/m_1/MaxPool
Node in deny list...delegated to ARM --- layer type - MaxPool, Node name - /model.9/m_2/MaxPool
INFORMATION -- [TIDL_ResizeLayer] Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION -- [TIDL_ResizeLayer] Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
Running runtimes graphviz - /home/mugu/edgeai-tidl-tools/tidl_tools/tidl_graphVisualiser_runtimes.out /home/mugu/edgeai-tidl-tools/model-artifacts/tidl_output/allowedNode.txt /home/mugu/edgeai-tidl-tools/model-artifacts/tidl_output/tempDir/graphvizInfo.txt /home/mugu/edgeai-tidl-tools/model-artifacts/tidl_output/tempDir/runtimes_visualization.svg In TIDL_createStateImportFunc Compute on node : TIDLExecutionProvider_TIDL_0_0 0, Conv, 3, 1, images, 0 1, Relu, 1, 1, 0, 1 2, Conv, 3, 1, 1, 2 3, Relu, 1, 1, 2, 3 4, Conv, 3, 1, 3, 4 5, Relu, 1, 1, 4, 5 6, Split, 1, 2, 5, 6 7, Conv, 3, 1, /model.2/Split_output_1, 7 8, Relu, 1, 1, 7, 8 9, Conv, 3, 1, 8, 9 10, Relu, 1, 1, 9, 10 11, Add, 2, 1, /model.2/Split_output_1, 11 12, Concat, 3, 1, 6, 12 13, Conv, 3, 1, 12, 13 14, Relu, 1, 1, 13, 14 15, Conv, 3, 1, 14, 15 16, Relu, 1, 1, 15, 16 17, Conv, 3, 1, 16, 17 18, Relu, 1, 1, 17, 18 19, Split, 1, 2, 18, 19 20, Conv, 3, 1, /model.4/Split_output_1, 20 21, Relu, 1, 1, 20, 21 22, Conv, 3, 1, 21, 22 23, Relu, 1, 1, 22, 23 24, Add, 2, 1, /model.4/Split_output_1, 24 25, Conv, 3, 1, 24, 25 26, Relu, 1, 1, 25, 26 27, Conv, 3, 1, 26, 27 28, Relu, 1, 1, 27, 28 29, Add, 2, 1, 24, 29 30, Concat, 4, 1, 19, 30 31, Conv, 3, 1, 30, 31 32, Relu, 1, 1, 31, 32 33, Conv, 3, 1, 32, 33 34, Relu, 1, 1, 33, 34 35, Conv, 3, 1, 34, 35 36, Relu, 1, 1, 35, 36 37, Split, 1, 2, 36, 37 38, Conv, 3, 1, /model.6/Split_output_1, 38 39, Relu, 1, 1, 38, 39 40, Conv, 3, 1, 39, 40 41, Relu, 1, 1, 40, 41 42, Add, 2, 1, /model.6/Split_output_1, 42 43, Conv, 3, 1, 42, 43 44, Relu, 1, 1, 43, 44 45, Conv, 3, 1, 44, 45 46, Relu, 1, 1, 45, 46 47, Add, 2, 1, 42, 47 48, Concat, 4, 1, 37, 48 49, Conv, 3, 1, 48, 49 50, Relu, 1, 1, 49, 50 51, Conv, 3, 1, 50, 51 52, Relu, 1, 1, 51, 52 53, Conv, 3, 1, 52, 53 54, Relu, 1, 1, 53, 54 55, Split, 1, 2, 54, 55 56, Conv, 3, 1, /model.8/Split_output_1, 56 57, Relu, 1, 1, 56, 57 58, Conv, 3, 1, 57, 58 59, Relu, 1, 1, 58, 59 60, Add, 2, 1, /model.8/Split_output_1, 60 61, Concat, 3, 1, 55, 61 62, Conv, 3, 1, 61, 62 63, Relu, 1, 1, 62, 63 64, Conv, 3, 1, 63, 64 65, Relu, 1, 1, 64, 65

Input tensor name - images Output tensor name - 32 Output tensor name - 50 Output tensor name - 65 In TIDL_createStateImportFunc Compute on node : TIDLExecutionProvider_TIDL_1_1 0, Concat, 4, 1, 65, 69 1, Conv, 3, 1, 69, 70 2, Relu, 1, 1, 70, 71 3, Resize, 3, 1, 71, 72 4, Concat, 2, 1, 72, 73 5, Conv, 3, 1, 73, 74 6, Relu, 1, 1, 74, 75 7, Split, 1, 2, 75, 76 8, Conv, 3, 1, /model.12/Split_output_1, 77 9, Relu, 1, 1, 77, 78 10, Conv, 3, 1, 78, 79 11, Relu, 1, 1, 79, 80 12, Concat, 3, 1, 76, 81 13, Conv, 3, 1, 81, 82 14, Relu, 1, 1, 82, 83 15, Resize, 3, 1, 83, 84 16, Concat, 2, 1, 84, 85 17, Conv, 3, 1, 85, 86 18, Relu, 1, 1, 86, 87 19, Split, 1, 2, 87, 88 20, Conv, 3, 1, /model.15/Split_output_1, 89 21, Relu, 1, 1, 89, 90 22, Conv, 3, 1, 90, 91 23, Relu, 1, 1, 91, 92 24, Concat, 3, 1, 88, 93 25, Conv, 3, 1, 93, 94 26, Relu, 1, 1, 94, 95 27, Conv, 3, 1, 95, 122 28, Relu, 1, 1, 122, 123 29, Conv, 3, 1, 123, 124 30, Relu, 1, 1, 124, 125 31, Conv, 3, 1, 125, 126 32, Conv, 3, 1, 95, 127 33, Relu, 1, 1, 127, 128 34, Conv, 3, 1, 128, 129 35, Relu, 1, 1, 129, 130 36, Conv, 3, 1, 130, 131 37, Concat, 2, 1, 126, 132 38, Reshape, 2, 1, 132, 155 39, Conv, 3, 1, 95, 96 40, Relu, 1, 1, 96, 97 41, Concat, 2, 1, 97, 98 42, Conv, 3, 1, 98, 99 43, Relu, 1, 1, 99, 100 44, Split, 1, 2, 100, 101 45, Conv, 3, 1, /model.18/Split_output_1, 102 46, Relu, 1, 1, 102, 103 47, Conv, 3, 1, 103, 104 48, Relu, 1, 1, 104, 105 49, Concat, 3, 1, 101, 106 50, Conv, 3, 1, 106, 107 51, Relu, 1, 1, 107, 108 52, Conv, 3, 1, 108, 133 53, Relu, 1, 1, 133, 134 54, Conv, 3, 1, 134, 135 55, Relu, 1, 1, 135, 136 56, Conv, 3, 1, 136, 137 57, Conv, 3, 1, 108, 138 58, Relu, 1, 1, 138, 139 59, Conv, 3, 1, 139, 140 60, Relu, 1, 1, 140, 141 61, Conv, 3, 1, 141, 142 62, Concat, 2, 1, 137, 143 63, Reshape, 2, 1, 143, 156 64, Conv, 3, 1, 108, 109 65, Relu, 1, 1, 109, 110 66, Concat, 2, 1, 110, 111 67, Conv, 3, 1, 111, 112 68, Relu, 1, 1, 112, 113 69, Split, 1, 2, 113, 114 70, Conv, 3, 1, /model.21/Split_output_1, 115 71, Relu, 1, 1, 115, 116 72, Conv, 3, 1, 116, 117 73, Relu, 1, 1, 117, 118 74, Concat, 3, 1, 114, 119 75, Conv, 3, 1, 119, 120 76, Relu, 1, 1, 120, 121 77, Conv, 3, 1, 121, 144 78, Relu, 1, 1, 144, 145 79, Conv, 3, 1, 145, 146 80, Relu, 1, 1, 146, 147 81, Conv, 3, 1, 147, 148 82, Conv, 3, 1, 121, 149 83, Relu, 1, 1, 149, 150 84, Conv, 3, 1, 150, 151 85, Relu, 1, 1, 151, 152 86, Conv, 3, 1, 152, 153 87, Concat, 2, 1, 148, 154 88, Reshape, 2, 1, 154, 157 89, Concat, 3, 1, 155, 158 90, Split, 1, 2, 158, 159 91, Reshape, 2, 1, 159, 160 92, Transpose, 1, 1, 160, 161 93, Softmax, 1, 1, 161, 162 94, Transpose, 1, 1, 162, 163 95, Conv, 2, 1, 163, 164 96, Reshape, 2, 1, 164, 165 97, Slice, 4, 1, 165, 166 98, Sub, 2, 1, /model.22/Constant_9_output_0, 168 99, Slice, 4, 1, 165, 167 100, Add, 2, 1, /model.22/Constant_9_output_0, 169 101, Add, 2, 1, 168, 170 102, Div, 2, 1, 170, 171 103, Sub, 2, 1, 169, 172 104, Concat, 2, 1, 171, 173 105, Mul, 2, 1, 173, 174 106, Sigmoid, 1, 1, /model.22/Split_output_1, 175 107, Concat, 2, 1, 174, output0

Input tensor name - 65

Input tensor name - 66

Input tensor name - 67

Input tensor name - 68

Input tensor name - 50

Input tensor name - 32 Output tensor name - output0 Input shape: [1, 3, 640, 640] Input "images": tensor(float) Graph Domain TO version : 11In TIDL_onnxRtImportInit subgraph_name=65 Layer 0, subgraph id 65, name=32 Layer 1, subgraph id 65, name=50 Layer 2, subgraph id 65, name=65 Layer 3, subgraph id 65, name=images In TIDL_runtimesOptimizeNet: LayerIndex = 70, dataIndex = 71

** Frame index 1 : Running float import ***** In TIDL_runtimesPostProcessNet In TIDL_runtimesPostProcessNet 1 In TIDL_runtimesPostProcessNet 2 In TIDL_runtimesPostProcessNet 3

ALL MODEL CHECK PASSED

In TIDL_runtimesPostProcessNet 4 **** in TIDL_subgraphRtCreate **** The soft limit is 2048 The hard limit is 2048 MEM: Init ... !!! MEM: Init ... Done !!! 0.0s: VX_ZONE_INIT:Enabled 0.3s: VX_ZONE_ERROR:Enabled 0.4s: VX_ZONE_WARNING:Enabled 0.1589s: VX_ZONE_INIT:[tivxInit:185] Initialization Done !!! **** TIDL_subgraphRtCreate done **** *** In TIDL_subgraphRtInvoke **** 0 1.00000 0.00000 1.00000 6 1 1.00000 0.00000 19.82178 6 2 1.00000 0.00000 16.53598 6 3 1.00000 0.00000 19.74176 6 4 1.00000 0.00000 19.74176 6 5 1.00000 0.00000 14.05759 6 6 1.00000 0.00000 14.51813 6 7 1.00000 0.00000 8.79035 6 8 1.00000 0.00000 15.49221 6 9 1.00000 0.00000 19.74176 6 10 1.00000 0.00000 16.90550 6 11 1.00000 0.00000 10.47098 6 12 1.00000 0.00000 10.40412 6 13 1.00000 0.00000 8.14298 6 14 1.00000 0.00000 10.40412 6 15 1.00000 0.00000 7.42323 6 16 1.00000 0.00000 6.29774 6 17 1.00000 0.00000 10.40412 6 18 1.00000 0.00000 7.00017 6 19 1.00000 0.00000 6.57213 6 20 1.00000 0.00000 12.55501 6 21 1.00000 0.00000 12.55501 6 22 1.00000 0.00000 8.02557 6 23 1.00000 0.00000 6.77668 6 24 1.00000 0.00000 7.29686 6 26 1.00000 0.00000 7.29686 6 27 1.00000 0.00000 6.49417 6 28 1.00000 0.00000 6.38711 6 29 1.00000 0.00000 7.75028 6 30 1.00000 0.00000 9.05462 6 31 1.00000 0.00000 7.86443 6 32 1.00000 0.00000 7.68957 6 33 1.00000 0.00000 12.36692 6 34 1.00000 0.00000 12.36692 6 35 1.00000 0.00000 7.27499 6 36 1.00000 0.00000 8.20982 6 37 1.00000 0.00000 6.06204 6 39 1.00000 0.00000 5.03477 6 40 1.00000 0.00000 6.06204 6 41 1.00000 0.00000 5.85117 6 42 1.00000 0.00000 6.02240 6 43 1.00000 0.00000 8.49440 6 44 1.00000 0.00000 8.49440 6 45 1.00000 0.00000 5.75977 6 46 1.00000 0.00000 6.35584 6 Layer, Layer Cycles,kernelOnlyCycles, coreLoopCycles,LayerSetupCycles,dmaPipeupCycles, dmaPipeDownCycles, PrefetchCycles,copyKerCoeffCycles,LayerDeinitCycles,LastBlockCycles, paddingTrigger, paddingWait,LayerWithoutPad,LayerHandleCopy, BackupCycles, RestoreCycles,

 1,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 2,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 3,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 4,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 5,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 6,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 7,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 8,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
 9,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
10,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
11,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
12,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
13,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
14,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
15,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
16,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
17,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
18,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
19,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
20,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
21,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
22,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
23,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
24,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,

26,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
27,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
28,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
29,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
30,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
31,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
32,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
33,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
34,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
35,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
36,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
37,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,

39,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
40,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
41,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
42,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
43,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
44,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
45,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,
46,              0,              0,              0,              0,              0,                 0,              0,                 0,              0,              0,              0,              0,              0,              0,              0,              0,

Sum of Layer Cycles 0 Sub Graph Stats 927.000000 6442865.000000 1928.000000 *** TIDL_subgraphRtInvoke done ****

** Frame Index 1 : Running float inference ** 2024-03-21 11:30:47.459985019 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running ReorderInput node. Name:'ReorderInput' Status Message: /home/kumar/work/ort_1.14/onnxruntime/onnxruntime/contrib_ops/cpu/nchwc_ops.cc:17 virtual onnxruntime::common::Status onnxruntime::contrib::ReorderInput::Compute(onnxruntime::OpKernelContext*) const X_rank == 4 was false.

Traceback (most recent call last): File "/home/mugu/edgeai-tidl-tools/compile_model.py", line 107, in sess.run(None, {input_name: input_data}) File "/home/mugu/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run return self._sess.run(output_names, input_feed, run_options) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running ReorderInput node. Name:'ReorderInput' Status Message: /home/kumar/work/ort_1.14/onnxruntime/onnxruntime/contrib_ops/cpu/nchwc_ops.cc:17 virtual onnxruntime::common::Status onnxruntime::contrib::ReorderInput::Compute(onnxruntime::OpKernelContext*) const X_rank == 4 was false.

**** in TIDL_subgraphRtDelete **** MEM: Deinit ... !!! MEM: Alloc's: 28 alloc's of 274886857 bytes MEM: Free's : 28 free's of 274886857 bytes MEM: Open's : 0 allocs of 0 bytes MEM: Deinit ... Done !!! **** in TIDL_subgraphRtDelete **** Segmentation fault (core dumped) ` My files generated in artifacts folder:

yolov8.zip

Thanks !

mathmanu commented 6 months ago

There are several layers at the end of a detection model that TIDL does not directly support. There are two methods to handle this: Option 1. Let the unsupported layers run on HostARM via onnxruntime. (slower execution time). Option 2. Handle the detection postprocessing layers in TIDL's DetectionPostProcess Layer (faster execution time).

Option 1 is possible even now. For option 2, YOLOV8 detection head will be supported in TIDL in 9.2 release (early next month).

mathmanu commented 6 months ago

Looping in the right expert who can take a look at the error that you got and provide insights. CC: @vtrip97

Mugutech62 commented 6 months ago

There are several layers at the end of a detection model that TIDL does not directly support. There are two methods to handle this: Option 1. Let the unsupported layers run on HostARM via onnxruntime. (slower execution time). Option 2. Handle the detection postprocessing layers in TIDL's DetectionPostProcess Layer (faster execution time).

Option 1 is possible even now. For option 2, YOLOV8 detection head will be supported in TIDL in 9.2 release (early next month). Thanks @mathmanu As for as now we have some urgent requirements, so we are going with option 1, can some of experts given some insights to solve this issue?

mathmanu commented 6 months ago

Will it be possible to share the onnx model?

Mugutech62 commented 6 months ago

Will it be possible to share the onnx model?

Yeah sure, you find the model in the zip file above Thanks

mathmanu commented 6 months ago

You mean the file yolov8.zip - there is no onnx file in that.

Mugutech62 commented 6 months ago

Hello @mathmanu We further optimised yolov8 by changing the maxpool kernal shape with 3*3, but failed to compile the model. The optimised model I have attached here with compilation logs: custom_model_logs.txt model: https://drive.google.com/file/d/1wpfgRop6tscEQcmDTO9dfvagiv3GJrge/view?usp=sharing Thank you

Mugutech62 commented 6 months ago

Hello @mathmanu @vmandela-ti Any updates on this issue?

Mugutech62 commented 6 months ago

Hello @mathmanu @vmandela-ti Is there any insights or advice from you side to solve this? Thank you

mathmanu commented 5 months ago

Hi @Mugutech62

We are in the process of publishing our fork of mmyolo, our trained lite models and model compilation options. We hope to complete it in a few days. Please give us a few days to complete it and you can take a look at it.

Mugutech62 commented 5 months ago

Hi @Mugutech62

We are in the process of publishing our fork of mmyolo, our trained lite models and model compilation options. We hope to complete it in a few days. Please give us a few days to complete it and you can take a look at it.

Hello @mathmanu Thanks for the reply, kindly take your time to finish of your work and reply us soon after looking at our model once Thank you

Mugutech62 commented 5 months ago

Hi @Mugutech62

We are in the process of publishing our fork of mmyolo, our trained lite models and model compilation options. We hope to complete it in a few days. Please give us a few days to complete it and you can take a look at it.

Hello @mathmanu and team Any updates regarding this model? My ti forum thread link: [https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1335845/tda4vm-custom-model-compilation-using-edgeai-tidl-tools/5130439?tisearch=e2e-sitesearch&keymatch=%20user%3A553216#5130439]()

mathmanu commented 5 months ago

The software is ready, but formal release will take a few more days. Let me share you the relevant files directly. Can you clarify again, which edge device are you working on?

Mugutech62 commented 5 months ago

The software is ready, but formal release will take a few more days. Let me share you the relevant files directly. Can you clarify again, which edge device are you working on?

Hello @mathmanu I am currently working on porting open source Yolov8 from ultralytics on edge device TDA4VM, I have made some changes in the model which I listed below: 1) SiLu to ReLu 2) Maxpool2d (5x kernal shape) ->Maxpool2D(3xkernal shape)

Mugutech62 commented 5 months ago

The software is ready, but formal release will take a few more days. Let me share you the relevant files directly. Can you clarify again, which edge device are you working on?

Can you please look out my ti forum post, where I have mentioned about the issue detaily

mathmanu commented 5 months ago

Attached a sample yolov8 lite model and training script changes in your e2e thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1335845/tda4vm-custom-model-compilation-using-edgeai-tidl-tools/5130439?tisearch=e2e-sitesearch

Also includes TIDL compilation options used in edgeai-benchmark in the config yaml file corresponding to thsi config: https://github.com/TexasInstruments/edgeai-benchmark/blob/main/configs/detection_v2.py#L169

Mugutech62 commented 5 months ago

Attached a sample yolov8 lite model and training script changes in your e2e thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1335845/tda4vm-custom-model-compilation-using-edgeai-tidl-tools/5130439?tisearch=e2e-sitesearch

Also includes TIDL compilation options used in edgeai-benchmark in the config yaml file corresponding to thsi config: https://github.com/TexasInstruments/edgeai-benchmark/blob/main/configs/detection_v2.py#L169

Thanks @mathmanu