Semantic Segmentation of Camera Input in Unity

[ranibu]

Do you support any CNN architectures for (real-time) semantic segmentation? We tried to import a number of state-of-the-art CNNs in ONNX format into Unity but all attempts proved to be futile.

[AlexRibard]

We do support a number of known architectures. https://github.com/onnx/models#image_classification a good number if not all of the models here imports and runs.

Could you clarify a bit more the issues you have ran into?

• Is it that your model doesn't import?
• Is it your model runs slow on the hardware you are targeting?
• Are you getting error at runtime?
• Is the model taking too much memory?
• ....

In any case, please provide your problematic model as well as the issue so we can investigate further. Thank you!

[ranibu]

Thanks for the quick reply!

To be more specific, one of the models of interest is PSPNet which I exported to ONNX (would you like me to provide you with the exported file?) The model doesn't import and Unity throws the following exception:

Currently only constant inputs for node of type Transpose are supported. Instead input of Transpose__68:0 is pointing to non-constant node SpaceToDepth__67:0.

Json: { "input": [ "SpaceToDepth__67:0" ], "output": [ "Transpose__68:0" ], "name": "Transpose__68", "opType": "Transpose", "attribute": [ { "name": "perm", "ints": [ "1", "0", "2", "3" ], "type": "INTS" } ] }
  at Unity.Barracuda.ONNXModelImporter.<.ctor>b__14_55 (Unity.Barracuda.ModelBuilder net, Unity.Barracuda.ONNXNodeWrapper node) [0x00145] in /Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:601 
  at Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) [0x0032f] in /Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1088 

Unity.Barracuda.ONNXModelImporter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1404)
Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1097)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1005)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at /home/bokken/buildslave/unity/build/Modules/AssetPipelineEditor/Public/ScriptedImporter.cs:22)

As a side note, we found that when trying to import from the segmentation networks https://github.com/onnx/models#object-detection--image-segmentation- such as Mask-RCNN they do not import.

[AlexRibard]

Ah yes, Mask-RCNN as still being worked on unfortunately... If you have the onnx model it would be very helpful thanks!

[ranibu]

Here you go https://easyupload.io/4wm84i thanks!

[AlexRibard]

ok I figured it out. This part messed up our importer. Can you make your input [1,1024,2048,3] instead of rank 3 [1024,2048,3]? That should fix the import problems while I fix the current model for our importer. Do send me the edited onnx model after, I'll validate that it works

Also judging with the error your are getting I think you might be on a old Barracuda version. I'd recommend you to take a more recent version, but out of curiosity which one are you on?

[AlexRibard]

Also be careful with your input range. For now it is within 0,255 Unity textures are in 0-1 range, so that might cause problem down the line

[ranibu]

Thank you so much! I am currently having trouble changing the input shape of our model but once it's fixed, I will report back to you.

We were using Barracuda version 1.0.4 because it is the latest verified version but now we updated it to version 2.0.0-pre4..

[ranibu]

@AlexRibard Thank you for your help so far, we have decided to work on a more promising architecture for our task (that is, DDRNet23 slim). For this reason, I am gonna close this issue.

[AlexRibard]

ok thanks