search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
17.19k stars 4.16k forks source link

Import of onnx file fails: Unexpected error while parsing layer 44 of type Gemm #4710

Closed simon-winter closed 3 years ago

simon-winter commented 3 years ago

Describe the bug Import of Onnx file fails with error "Unexpected error while parsing layer 44 of type Gemm" when using a custom Sensor implementation for oberserving a float 2D grid making use of the CNN.

The GridSensor implementation can be found here: https://github.com/mbaske/grid-sensor/tree/master/GridSensor To use it:

To Reproduce Steps to reproduce the behavior:

  1. Use GridSensor as described or get the broken onnx here: bugRepOnnx.zip 1.1 train a few seconds, only when not using the file)

  2. try importing saved model/file into Unity

  3. See errors

Console logs / stack traces

Assertion failed
UnityEngine.Debug:Assert(Boolean)
Unity.Barracuda.ONNXModelImporter:SwapSpatialDimensionsAndFeaturesInMatMulWeights(Tensor, Int32, Layout) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:841)
Unity.Barracuda.ONNXModelImporter:<.ctor>b__14_58(ModelBuilder, ONNXNodeWrapper) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:623)
Unity.Barracuda.ONNXModelImporter:ConvertOnnxModel(ModelProto) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1088)
Unity.Barracuda.ONNXModelImporter:OnImportAsset(AssetImportContext) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1005)
UnityEditor.Experimental.AssetImporters.ScriptedImporter:GenerateAssetData(AssetImportContext)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)
OnnxImportException: Unexpected error while parsing layer 44 of type Gemm.
Assertion failure. Values are not equal.
Expected: 71680 == 83456

Json: { "input": [ "43", "network_body.linear_encoder.seq_layers.0.weight", "network_body.linear_encoder.seq_layers.0.bias" ], "output": [ "44" ], "name": "Gemm_9", "opType": "Gemm", "attribute": [ { "name": "alpha", "f": 1, "type": "FLOAT" }, { "name": "beta", "f": 1, "type": "FLOAT" }, { "name": "transB", "i": "1", "type": "INT" } ] }
  at UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) [0x00043] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) [0x0005f] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) [0x00009] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) [0x0000c] in <b53f6837a4614744b595be69da4cbb29>:0 
  at Unity.Barracuda.Tensor.Reshape (Unity.Barracuda.TensorShape newShape, System.String newName) [0x00001] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Runtime\Core\Tensor.cs:810 
  at Unity.Barracuda.ONNXModelImporter.SwapSpatialDimensionsAndFeaturesInMatMulWeights (Unity.Barracuda.Tensor weights, System.Int32 featureCount, Unity.Barracuda.VariableTensor+Layout layout) [0x00051] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:843 
  at Unity.Barracuda.ONNXModelImporter.<.ctor>b__14_58 (Unity.Barracuda.ModelBuilder net, Unity.Barracuda.ONNXNodeWrapper node) [0x00074] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:623 
  at Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) [0x00367] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:1088 

Unity.Barracuda.ONNXModelImporter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1404)
Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1097)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1005)
UnityEditor.Experimental.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at <b5e754b172ef4f1b82e78043441796df>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

Asset import failed, "Assets/explorer.onnx" > OnnxImportException: Unexpected error while parsing layer 44 of type Gemm.
Assertion failure. Values are not equal.
Expected: 71680 == 83456

Json: { "input": [ "43", "network_body.linear_encoder.seq_layers.0.weight", "network_body.linear_encoder.seq_layers.0.bias" ], "output": [ "44" ], "name": "Gemm_9", "opType": "Gemm", "attribute": [ { "name": "alpha", "f": 1, "type": "FLOAT" }, { "name": "beta", "f": 1, "type": "FLOAT" }, { "name": "transB", "i": "1", "type": "INT" } ] }
  at UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) [0x00043] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) [0x0005f] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) [0x00009] in <b53f6837a4614744b595be69da4cbb29>:0 
  at UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) [0x0000c] in <b53f6837a4614744b595be69da4cbb29>:0 
  at Unity.Barracuda.Tensor.Reshape (Unity.Barracuda.TensorShape newShape, System.String newName) [0x00001] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Runtime\Core\Tensor.cs:810 
  at Unity.Barracuda.ONNXModelImporter.SwapSpatialDimensionsAndFeaturesInMatMulWeights (Unity.Barracuda.Tensor weights, System.Int32 featureCount, Unity.Barracuda.VariableTensor+Layout layout) [0x00051] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:843 
  at Unity.Barracuda.ONNXModelImporter.<.ctor>b__14_58 (Unity.Barracuda.ModelBuilder net, Unity.Barracuda.ONNXNodeWrapper node) [0x00074] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:623 
  at Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) [0x00367] in C:\Users\Sebi\Documents\GitHub\MLCleaningRobot\Library\PackageCache\com.unity.barracuda@1.0.4\Barracuda\Editor\ONNXModelImporter.cs:1088 

Unity.Barracuda.ONNXModelImporter.Err (Unity.Barracuda.Model model, System.String layerName, System.String message, System.String extendedMessage, System.String debugMessage) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1404)
Unity.Barracuda.ONNXModelImporter.ConvertOnnxModel (Onnx.ModelProto onnxModel) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1097)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/com.unity.barracuda@1.0.4/Barracuda/Editor/ONNXModelImporter.cs:1005)
UnityEditor.Experimental.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.Experimental.AssetImporters.AssetImportContext ctx) (at <b5e754b172ef4f1b82e78043441796df>:0)
UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

UnityEditorInternal.InternalEditorUtility:ProjectWindowDrag(HierarchyProperty, Boolean)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

Screenshots the output when starting a new training has also some warnings in there. bugrep1

Environment (please complete the following information):

simon-winter commented 3 years ago

Note i could reproduce this with the https://github.com/Unity-Technologies/ml-agents/blob/gridworld-custom-obs/Project/Assets/ML-Agents/Examples/GridWorld/Scripts/GridSensorComponent.cs

vincentpierre commented 3 years ago

I seem to be able import the explorer.onnx file just fine on both Unity 2018.4 and 2020.1 when using ML-Agents release 10. I notice you are on Barracuda version 1.0.4 from the error messages you have. There is also a warning message at the start of training that says that you have different versions of ML-Agents in Unity and C#. I think you are using ML-Agents package 1.0.6 in Unity which is not compatible with release 10 of the trainers (trainers version 0.22.0). It says on the documentation or package 1.0.6 of the package that it is "strongly recommended to use trainers version 0.16.0 but trainers up to trainer version 0.20.0 are supported".

To resolve compatibility, you need to either upgrade the MLAgents Unity package to a more recent (but preview) version (such as 1.6.0) in the Unity package manager OR downgrade the Python trainers to at least version 0.20.0 (pip install mlagents==0.20.0).

The versions of ML-Agents can be confusing, there is a doc here to clarify some things

simon-winter commented 3 years ago

Thank you! I didnt expand the preview section and took the latest stable which was 1.0.6, the not matching one. Taking the most recent preview version and the error is gone.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.