NOTSET padding of ONNX

[fmannhardt]

I am trying to convert a ONNX model to the Barracuda format. My model uses NOTSET for the attribute auto_pad: https://github.com/onnx/onnx/blob/master/docs/Operators.md#conv

According to the ONNX documentation that means that the padding is manually specified in the attribute pad. I assume that what it would be needed to copy the values in pad as done when auto_pad is not present, right?

So, a fix would be to include auto_pad == "NOTSET" in the check for absence of auto padding?

[FlorentGuinier]

Hey, fmannhardt NOTSET in autopad works as expected in Barracuda 0.3 via the new ONNX importer.

To use the new importer just drag ONNX file into Unity :) See https://github.com/mantasp/barracuda-release/blob/release/0.3.0/Documentation~/Barracuda.md

Have a great day!

[fmannhardt]

Thanks, I am trying the new release now, but ONNX files won't be auto converted when dragged into Unity. I am fairly new to Unity. Is there any magic configuration one needs to do for that?

I installed the Barracuda package via package manager and I am using Unity 2018.4.12f1

[mantasp]

@fmannhardt what do you see when click on ONNX model file in Unity?

[fmannhardt]

Nothing special, it looks like this:

I installed the package through the "Package Manager" and updated to v0.3.1, but no changes.

Is there a plan to further support/update the converters in Python? In my case our ML pipeline is all based on Python code, so it would be more natural to have the conversion there. However, I have seen multiple issues (Keras converter does not know about activation functions as layers like Relu/Leakyrelu, ONNX converter seems to mess up the shapes of the output layer in some cases)

[mantasp]

That's quite weird, could you please double check that you don't have another instance of Barracuda in your project? also please check Package/manifest.json if it actually specifies Barracuda 0.3.x? If none of these shows issue, could you please share your model with us?

[fmannhardt]

I think I found the issue, I am using Assembly Definitions and I tried to drag the ONNX files to a 'StreamingAssets' folder which was not part of any Assembly Definition. I tried to drag the file into a folder that is part of an Assembly Definition and the model was converted.

Do you have an answer on whether the Python converted are going to be maintained or if there a plan to make the C# converter available standalone? My use case would be to have the conversion as part of a Machine Learning pipeline, so Unity would not be available.

Also, I noted that the ONNX file is left unchanged and you recommend to assign it via the Unity assets mechanism (public field). Is there a way to save the NN file?

[mantasp]

Aha! StreamingAsset folder has special meaning, as it forces Unity Editor to skip import pipeline and embeds these assets as a raw files. Any other folder will work fine for ONNX importer. For ONNX C# importer is the future and python version will be deprecated. While in editor ONNX file remains unchanged, but if you do standalone build of your app it will be internally serialised to NN file. Is there any reason you want to access it directly? Recommended workflow is to have NNModel public field in your MonoBehviour script and drag ONNX model into that field, all the serialisation to NN file and dependency tracking will happen behind the scenes.. Then you can pass NNModel object to Barracuda ModelLoader.

[fmannhardt]

Thanks, I am new to Unity and I was not aware of the special handling for that folder.

While in editor ONNX file remains unchanged, but if you do standalone build of your app it will be internally serialised to NN file. Is there any reason you want to access it directly? Recommended workflow is to have NNModel public field in your MonoBehviour script and drag ONNX model into that field, all the serialisation to NN file and dependency tracking will happen behind the scenes.. Then you can pass NNModel object to Barracuda ModelLoader.

Our runtime needs to handle multiple models that are dynamically loaded. So at design time the actual model is not known. Also, for tests there different models may be required and the classes running the NN inference related code are encapsulated so don't necessarily need to know about Unity / MonoBehaviour.

How would that work with the recommended workflow?

Having the conversion outside of Unity would make sure that we know in our ML pipeline whether the model can be successfully converted. A standalone C# converter would be also fine for that but to require Unity would be inconvenient as this seems to be difficult for standard CI/CD pipelines.

[mantasp]

Recommended workflow depends on your final use cases. Could you please tell little bit more about your use case? Are you building Unity Standalon builds and deploying them to some target device?

[fmannhardt]

The use case is simply being able to dynamically deploy neural networks in our software. So, during build time, for example, for Android or UWP, there is only a base model included (or none at all) and additional networks will be loaded at run-time based on the use case or based on license etc. Also, model may need to be re-trained and updated without re-installing the whole application.

I don't see how the add it as "asset" and link it in the Unity editor workflow would support this use case. Does this make sense?

[mantasp]

It sounds that Unity Addressables would be best fit for your use case: https://docs.unity3d.com/Packages/com.unity.addressables@1.5/manual/AddressableAssetsGettingStarted.html It allows packaging and distribution of the assets separately from updating main application.

[fmannhardt]

Thanks. I will look at this. Nevertheless, for us it would be great to have the simple alternative of being able to convert them outside of Unity and without having to build Assets.