ArgumentException: Off-axis dimensions must match

[ameliesophie]

Hello,

I am trying to incorporate an ONNX model into an AR App based on ARFoundation in order to apply a semantic segmentation on the camera input. Therefore I exported this model to ONNX: https://github.com/ydhongHIT/DDRNet. The exported file and a visualization of the architecture can be found here: DDRNet.zip

I tested the exported model via onnxruntime on a random input and got an ouput of dimension [1,19,64,128] as expected. Importing the model to Unity was successful as well, but when I run the app I get the following error:

ArgumentException: Off-axis dimensions must match 05-29 19:34:11.491 12970 13086 E Unity : at Unity.Barracuda.TensorExtensions.Concat (Unity.Barracuda.TensorShape[] shapes, System.Int32 axis) [0x0008d] in <85103926d60f417ea609911b6e59fbcc>:0 05-29 19:34:11.491 12970 13086 E Unity : at Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary2[TKey,TValue] inputShapes, System.Collections.Generic.IDictionary2[System.String,System.Nullable1[Unity.Barracuda.TensorShape]]& shapesByName) [0x00c7a] in <85103926d60f417ea609911b6e59fbcc>:0 05-29 19:34:11.491 12970 13086 E Unity : at Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary2[TKey,TValue] inputShapes) [0x00000] in <85103926d60f417ea609911b6e59fbcc>:0 05-29 19:34:11.491 12970 13086 E Unity : at Unity.Barracuda.ModelAnalyzer.FindLargestNecessaryTensorShape (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes) [0x00000] in <85103926d60f417ea609911b6e59fbcc>:0 05-29 19:34:11.491 12970 13086 E Unity : at Unity.Barracuda.GenericVarsWithPreallocation.PrepareStorage (Un

I have read in older issues that there were some Barracuda internal problems with the Concat method, should this already be fixed in Barracuda version 2.0.0-pre.4? Or could this error relate to that as well? Do you have any advice how to circumvent this issue in this case?

I highly appreciate any help, thanks in advance!

[AlexRibard]

Hi @ameliesophie thank you for providing the model, that is very helpful. I've tested on latest, the model imports and runs. So that's good :) However I am getting some discrepancy when comparing against onnxruntime. We'll spend some time looking into this this week.

Out of curiosity, on which version are you?

[ameliesophie]

Thank you very much!

I am currently using Barracuda version 2.0.0-pre.4 in Unity 2020.3.4f1.

[AlexRibard]

I tested in 2.0.0, it actually runs fine with input shape 1,512,1024,3 The discrepancy comes from onnxruntime Upsample.linear mode. We default to bilinear and there was some issues between onnxruntime/pytroch and tf regarding bilinear upsampling. So it's probably a none issue. Using nearest is fine however.

[AlexRibard]

What input shape are you using?

[ameliesophie]

Indeed, the input shape I used was incorrect and caused the error I got. With the correct input shape the model runs in my project as well now.

Is there a way to change the Upsample mode to nearest in Unity or do I have to change the ONNX model?

[palder]

Hi there!

I get the same error as ameliesophie when I try to import my ONNX file into Barracuda. Is the input shape the issue here as well?

What I'm trying to accomplish

Object Detection using Barracuda and ARFoundation using a Custom Dataset. At this point I'm only using a single class. The annotations were created using roboflow.com and for training I used darknet , pre-trained weights darknet53.conv.74and a yolov3_tiny.cfg configuration. Then I converted the model to a frozen TensorFlow model. The conversion to ONNX was done with tf2onnx using the following command

python3 -m tf2onnx.convert --graphdef frozen_tf_model.pb --inputs=inputs:0 --outputs=output_boxes:0 --output model.onnx

ONNX model properties (info from netron.app):

Input: inputs:0 Type: float32[unk__143,416,416,3]

Output: output_boxes:0 Type: float32[unk__144,2535,6]

Problem

Whenever I import the ONNX file into Unity (using Barracuda 2.0.0-pre.5) , I get the following error:

Asset import failed, "Assets/Models/model5 1.onnx" > ArgumentException: Off-axis dimensions must match
Unity.Barracuda.TensorExtensions.Concat (Unity.Barracuda.TensorShape[] shapes, System.Int32 axis) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/TensorExtensions.cs:345)
Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes, System.Collections.Generic.IDictionary`2[System.String,System.Nullable`1[Unity.Barracuda.TensorShape]]& shapesByName) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/Backends/ModelAnalyzer.cs:472)
Unity.Barracuda.Compiler.Passes.NCHWToNHWCPass.CorrectOutputLayoutToMatchNHWCLayout (Unity.Barracuda.Model& nhwc) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/Compiler/Passes/NCHWToNHWC/CorrectOutputLayoutToMatchNHWCLayout.cs:19)
Unity.Barracuda.Compiler.Passes.NCHWToNHWCPass.Rewrite (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/Compiler/Passes/NCHWToNHWCPass.cs:160)
Unity.Barracuda.Compiler.Passes.NCHWToNHWCPass.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/Compiler/Passes/NCHWToNHWCPass.cs:37)
Unity.Barracuda.Compiler.Passes.IntermediateToRunnableNHWCPass.Run (Unity.Barracuda.Model& model) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/Core/Compiler/Passes/IntermediateToRunnableNHWCPass.cs:33)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:170)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:83)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@2.0.0-pre.5/Barracuda/Editor/ONNXModelImporter.cs:58)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at /Users/bokken/buildslave/unity/build/Modules/AssetPipelineEditor/Public/ScriptedImporter.cs:22)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&) (at /Users/bokken/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:189)

UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

Versions

Unity: 2020.3.10f1 Barracuda: 2.0.0-pre.5 tf2onnx: 1.9.0

Please let me know if I can provide anymore information or be of any help.

I mainly followed theses guides with some adaptations:

https://github.com/derenlei/Unity_Detection2AR https://medium.com/analytics-vidhya/yolov3-custom-object-detection-with-transfer-learning-47186c8f166d https://medium.com/analytics-vidhya/yolov3-to-tensorflow-lite-conversion-4602cec5c239

[AlexRibard]

@palder could you provide your onnx model?

Also what input shape are you using in Barracuda? Do note that Barracuda is in NHWC compared to onnx in NCHW https://github.com/Unity-Technologies/barracuda-release/blob/release/1.0.0/Documentation%7E/TensorHandling.md For for example, in @ameliesophie case the input in onnx is N:1, C:3, H:512, W:1024 so in Barracuda I need to construct my input tensor as follows N:1, H:512, W:1024, C:3

Let me know

[palder]

I changed the input of the ONNX model from NHWC to NCHW. Both of the models gave me same ArgumentException: Off-axis dimensions must match upon import. Here are my models: model_nchw.onnx model_nhwc.onnx

[AlexRibard]

Oh sorry if my explanation was not clear. I meant in Barracuda, we are in NHWC. Keep your onnx model in NCHW, we do the conversion at import time. And then in Barracuda construct your input tensor to be NHWC

I will try your models

[AlexRibard]

@palder I checked your model. It sadly doesn't import at the moment. Checking why

[ranibu]

@AlexRibard Following up on ameliesophie's question, how do you solve this?

Is there a way to change the Upsample mode to nearest in Unity or do I have to change the ONNX model?

The above was asked in response to this reply:

The discrepancy comes from onnxruntime Upsample.linear mode. We default to bilinear and there was some issues between onnxruntime/pytroch and tf regarding bilinear upsampling. So it's probably a none issue. Using nearest is fine however.

[AlexRibard]

I would first test out that the result it's producing is not desirable. We have a lot of users using Bilinear and it is fine. So do double check that the output is not what you want. (We do match Resize for example) so again, it might be a none issue.

If you do want to change there is two choices:

• edit the onnx file directly as so

  
  import onnx

model = onnx.load("mymodel.onnx")

graph = model.graph for node in graph.node: if node.op_type in ['Upsample']: for i in range(len(node.attribute)): if(node.attribute[i].name == "mode"): node.attribute[i].s = b'nearest'

onnx.save(model, "mymodel.onnx")

- modify the model once it is in Unity.

for (int i = 0; i < model.layers.Count; i++) Layer l = model.layers[i] if(l.type = Layer.Type.Upsample2D) model.layers[i].axis = -1

[AlexRibard]

@palder I have found the fix for your model. Pushing it. It won't make it in the upcoming version however. So you might have to wait a bit. You can email me at barracuda-support@unity3d.com I can send you the PR

[ameliesophie]

I would first test out that the result it's producing is not desirable. We have a lot of users using Bilinear and it is fine. So do double check that the output is not what you want. (We do match Resize for example) so again, it might be a none issue.

If you do want to change there is two choices:

* edit the onnx file directly as so

import onnx

model = onnx.load("mymodel.onnx")

graph = model.graph
for node in graph.node:
    if node.op_type in ['Upsample']:
        for i in range(len(node.attribute)):
            if(node.attribute[i].name == "mode"):
                node.attribute[i].s = b'nearest'

onnx.save(model, "mymodel.onnx")

* modify the model once it is in Unity.

for (int i = 0; i < model.layers.Count; i++)
   Layer l = model.layers[i]
   if(l.type = Layer.Type.Upsample2D)
      model.layers[i].axis = -1

Sorry, now I understand that you meant by a none issue. I get good results with bilinear upsampling, too. Thanks for your support!