Yolo-v3-tiny ArgumentException: Off-axis dimensions must match

[ROBYER1]

When using this repo for AR Foundation + Barracuda https://github.com/derenlei/Unity_Detection2AR

I used AlexeyAB's Darknet fork to train a custom model using Yolo-v3-tiny making my own weights from scratch to make a detector for apples/oranges as a quick test.

I converted the model to a tensorflow graph then an ONNX file using your instructions, making sure to follow the rules for Yolo v3 Tiny, there were no errors.

I pulled the ONNX file and labels into the project to test, making sure to edit Detector.cs to have: private const string INPUT_NAME = "yolov3-tiny/net1"; private const string OUTPUT_NAME = "yolov3-tiny/convolutional10/BiasAdd";

What was interesting was my ONNX file has 2 outputs but you only have a variable for one output, not sure why I need 2 outputs though...

At runtime I get this error: ArgumentException: Off-axis dimensions must match Unity.Barracuda.TensorExtensions.Concat (Unity.Barracuda.TensorShape[] shapes, System.Int32 axis) (at Library/PackageCache/com.unity.barracuda@1.2.0-preview/Barracuda/Runtime/Core/TensorExtensions.cs:338)

I have uploaded the graph and label file here Uploaded here if it helps to have a look, I'm pretty stumped but my suspicions are:

1.My yolo model had 2 outputs for some reason rather than 1 (I didn't change anything to make that, perhaps Yolo v3 tiny has 2 outputs rather than one which Yolo v2 tiny has.

2.The onnx file created could be a different format to the Unity default NHWC in which case Unity Barracuda must convert it on import which could flatted or interfere with some values?

[AlexRibard]

Hi @ROBYER1 ! Your model is perfectly valid, even if it has two outputs it's good. From what I can see, we must have a import bug on our side and wrongly interpret the axis in Concat. I'll fix it. If the fix is short enough I'll send you what code modification you need.

[ROBYER1]

Hi @ROBYER1 ! Your model is perfectly valid, even if it has two inputs it's good. From what I can see, we must have a import bug on our side and wrongly interpret the axis in Concat. I'll fix it. If the fix is short enough I'll send you what code modification you need.

Thanks Alex, and did you mean Outputs not inputs there? I would prefer if my model only had one output as the example code in the Detection2AR just queries one Output with the worker.

Is the fact that my model has 2 outputs just completely coincidental? I made it from scratch in darknet, freshly made weights so any tips to limit a model to one output would be appreciated too!

[AlexRibard]

Sorry, typo. Yes I meant it is ok if it has two outputs. No sure why your exported model has two outputs. My guess is that it depends on the implementation. But now that I think of it, you say that your model is a Yolo-v3 If that is the case it's missing the bottom part of the model doing the classification. The output of this model is still a image. Looks like you exported only the top part of the Yolo model... Not sure if this is intentional?

[ROBYER1]

Sorry, typo. Yes I meant it is ok if it has two outputs. No sure why your exported model has two outputs. My guess is that it depends on the implementation. But now that I think of it, you say that your model is a Yolo-v3 If that is the case it's missing the bottom part of the model doing the classification. The output of this model is still a image. Looks like you exported only the top part of the Yolo model... Not sure if this is intentional?

Ah wasn't intentional in that case, before conversion from Darknet Yolo-v3-Tiny weights to Tensorflow graph to ONNX, I checked using Darknet that the model weights worked as expected. It worked perfectly at detecting Apples/Oranges with unique test images.

I can only assume that something happened during the conversion, however I had no errors during and the file looked fine. I have no way of testing the ONNX file if Unity isn't importing it properly though.

Is there a good quick way to test the model as a Tensorflow Graph once converted then also as the ONNX file once I have converted the TF graph to ONNX?

I will investigate to see what stage of the conversion does that during the conversion from Darknet weights - Tensorflow Graph - ONNX file.

[AlexRibard]

Ok so to validate the import pipeline here is what we recommend.

1* follow this guide on exporting your TF model to Onnx https://github.com/Unity-Technologies/barracuda-release/blob/release/1.0.0/Documentation%7E/Exporting.md

2* to validate that your onnx model is indeed working what we do is we generate a random input and feed that to both onnx and tensorflow and check the output To execute a onnx file, you can use onnx-runtime (https://github.com/microsoft/onnxruntime) they have a python API so it's straightforward. You also compare TF .pb files with the onnx file using NETRON (https://github.com/lutzroeder/netron)

3* import in unity, make sure it imports without error

4* validate that it works by checking that it gives you the same value as onnx with that random input generated above

Let me know

[ROBYER1]

Ok so to validate the import pipeline here is what we recommend.

1* follow this guide on exporting your TF model to Onnx https://github.com/Unity-Technologies/barracuda-release/blob/release/1.0.0/Documentation%7E/Exporting.md

2* to validate that your onnx model is indeed working what we do is we generate a random input and feed that to both onnx and tensorflow and check the output To execute a onnx file, you can use onnx-runtime (https://github.com/microsoft/onnxruntime) they have a python API so it's straightforward. You also compare TF .pb files with the onnx file using NETRON (https://github.com/lutzroeder/netron)

3* import in unity, make sure it imports without error

4* validate that it works by checking that it gives you the same value as onnx with that random input generated above

Let me know

Thanks, the onnx-runtime link doesn't seem to explain how to execute a .onnx file anywhere, I want to feed it an image of an orangr or apple to validate it works as expected but can't see a good tutorial or starting point as to how to do that?

[AlexRibard]

https://github.com/microsoft/onnxruntime/blob/master/docs/python/tutorial.rst

import numpy
import onnxruntime as rt

sess = rt.InferenceSession(modelName)
input_name = sess.get_inputs()[0].name
input = np.array((1,3,416,416)) // or load the image into a numpy array
pred_onx = sess.run(None, {input_name: input.astype(numpy.float32)})[0]
print(pred_onx)

Yeah than you make sure the inferred class number matches.

[ROBYER1]

@AlexRibard thanks that was easy, I get this error when running the onnxruntime test

onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid rank for input: yolov3-tiny/net1:0 Got: 1 Expected: 4 Please fix either the inputs or the model.

Doubled back and check the darknet weights using darknet detector test, works as expected from the original weights so I can only assume that something goes wrong when converting from Darknet Yolo v3 tiny weights to TF using DW2TF

These are the instructions I used to convert the Darknet Yolo v3 Tiny custom weights - TF - ONNX

1. Install TensorFlow for python. Use the DW2TF repo on github (https://github.com/jinyu121/DW2TF). In my case, I'm using the command for yolov3-tiny. After this step, you will have both tensorflow weights and graph.
2. Then you need to figure out what is the input and output node names for the converted TF graph. It's easy to google that two to three lines of python.
3. Freeze the tf graph to convert model parameters into constants. Something like this: python3 -m tensorflow.python.tools.freeze_graph --input_graph=./checkpoint_dir/yolov3-tiny-food.pb --input_binary=true --output_node_names=yolov3-tiny/convolutional9/BiasAdd --input_checkpoint=./checkpoint_dir/yolov3-tiny-food.ckpt --output_graph=./checkpoint_dir/yolov3-tiny-food-freeze.pb Modify these paths and node names and I guess it will work for you as well.
4. convert tf model to onnx. You can find the relevant scripts here: https://github.com/onnx/tensorflow-onnx The command I previously used is: python3 -m tf2onnx.convert --graphdef ./yolov3-tiny-food-freeze.pb --output yolov3-tiny-food-freeze.onnx --inputs yolov3-tiny/net1:0 --outputs yolov3-tiny/convolutional9/BiasAdd:0

I have the Darknet weights, converted TF graph, frozen TF graph and ONNX model in the file here https://drive.google.com/file/d/1S5xHbgDL0TRmlsDjyEFrFXey8Jmt-iN9/view?usp=sharing

[AlexRibard]

How are you creating the input to onnx-runtime?

inputsTensor = np.random.rand(1,416,416,3)
inputsTensor = inputsTensor.astype(np.float32)

Looks like you passed in a rank 1 tensor. I would double check the shape of your loaded image. Also there seems to have a problem in how you froze/exported the .pb or .onnx they both are truncated at the bottom for some reason. Here is the model you should be getting in the end https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/yolov4

[AlexRibard]

I found this github that seems to export correctly a darknet yolo v4 model to onnx. Might inspire you https://github.com/Tianxiaomo/pytorch-YOLOv4

https://github.com/Tianxiaomo/pytorch-YOLOv4/blob/master/demo_darknet2onnx.py

But he seems to go via pytorch. Might be easier...

[ROBYER1]

Here is my project I am using to test giving the onnx input of the camera image (simulated) in the editor through AR Simulation Package. I just have a plane with a picture of the object to be detected in the AR scene for the editor, you might find it useful for internal testing perhaps.

https://drive.google.com/file/d/1DKfwQYaL2OIIKdYYi2fUeNYURiKK9yUj/view?usp=sharing

I recorded a quick vid of how it looks, showing the v2 tiny model working fine but the v3 tiny does not https://drive.google.com/file/d/19AKUCu65eHJqKSyw9dJd_mbYyQVPkldJ/view?usp=sharing

If you wanted to use it for testing models, follow what I do in the video which is:

1. In Scenes/Detect - On the detector script, assign .names and .onnx file, set input and output names to match those on the .onnx file in the inspector

2. On Test/DetectorTestPlane, swap out the image on the material for the image you want to detect

3. The Prefab plane is simulated in the scene so that it acts as an AR plane, you need to remove then re-add it in the scene to make it refresh the new image by going to ARSim - Detector(script) - Model File variable - remove/re-add the prefab

I want to try out the pytorch conversion example, however it requires TensorRT which doesn's seem to have a version for Ubuntu 20.04 and the Windows version of TensorRT doesn't have python support (I am running Ubuntu 20.04 on windows through WSL 2.0 for my workflows rather than a dual boot).

[ROBYER1]

I tried the pytorch converter, it has an unresolved issue converting yolo v3 tiny https://github.com/Tianxiaomo/pytorch-YOLOv4/issues/313

Feels like it is difficult to convert to .onnx files :(

[ROBYER1]

@AlexRibard the converter does work for Yolo V4 Tiny however, which leaves me in an awkward spot unless Unity Barracuda gets support for Yolo V4 Tiny models.

I may have to fall back to using Yolo V2 Tiny (which has lower performance) due to the converters for Yolo V3 Tiny not working correctly

[ROBYER1]

I have a proper .onnx file for my yolo v3 tiny model here, the nodes are perfect this time but having trouble getting a result https://drive.google.com/file/d/1uY81Gh2Edh63R7G_6i5KI_oVLdMCuiem/view?usp=sharing

[AlexRibard]

Sorry, I must be missing something : here the end of the latest YoloV3-tiny model you shared to me Is the output what you want? I would have imagine it would output the inferred class of the image?

[AlexRibard]

Can you also share your YoloV4 model? I'll check what is missing to make it work.

[AlexRibard]

@ROBYER1 I've tested the model you shared me (yolov3-tiny-darkmarktest-416.onnx) on version 1.2.1 and validated it against onnx-runtime for a random input. I get correct results for both outputs at 10^-5 (absolute and relative error) I'm checking with the project you gave me

[ROBYER1]

@ROBYER1 I've tested the model you shared me: yolov3-tiny-darkmarktest-416.onnx and validated it against onnx-runtime for a random input. I get correct results for both outputs at 10^-5 (absolute and relative error)

Yes that onnx is the one, I believe my issue in the project is the detection script in the Detector scene is made for a Yolo-v2-tiny model, I think the code may be difficult to adapt for Yolo-v3-tiny. The parser for Yolo-v2-tiny is made from this Microsoft C# example but they haven't updated the example to Yolo-v3-tiny.

From your end I just want to make sure the model is reading correct in Unity, I think that figuring out how to adapt the Detector.cs script in that project from Yolo-v2-tiny to Yolo-v3-tiny is too difficult a task for me given the lack of C# reference in doing it, I may fall back to working with Yolo-v2-tiny. On that note, will there be some Unity examples added for barracuda for the use of Yolo object detection? Would be really handy for getting started with Barracuda.

I was sure the file wasn't truncated as it looked fine in Netron when I checked the graph, in the image below, on the left is the original Yolo-v3-tiny.onnx file I converted from AlexeyAb , the middle is my own test model .onnx file using a data set for recognising apples/oranges (as a test) and on the right is the .cfg of my test model to check that nothing is being missed out.

[ROBYER1]

Can you also share your YoloV4 model? I'll check what is missing to make it work.

@AlexRibard I can train and convert a Darknet Yolov4-tiny-416 model to .onnx tomorrow to send over as an example for future Yolo v4 tiny support if that helps.

I also hope that the project I sent over is useful, please let me know if you have the project now as I will probably need to remove the gdrive link for privacy after.

[AlexRibard]

Sorry, yeah my bad. I got confused with the outputs, I was thinking of mobilenet... Yes the outputs are correct. I did double check the model present in the repo yolov2-tiny-food-freeze.onnx and this too passes the onnx validation. So on my end, this looks like the problem lies in interpreting the output data from the model. I personally do not know how the classes/bboxes are encoded for yolo-v3, Judging on what I see online, I think it highly depends on the model tbh.

I have found a good example on the onnxmodel zoo page https://github.com/onnx/models#object_detection https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/tiny-yolov3 Maybe you can get inspired by it...

As regards to the tutorials, yes it is something we are looking to do, but right now our bandwidth is low unfortunately

For the yolo-v4, no need to train it, random weights is fine.

I'm closing the thread as the two models you provided passes validation on 1.2.1 if that is ok with you?

But let's follow up on this offline, here our support email: barracuda-support@unity3d.com, feel free to email us.

P.S: I have downloaded your repo, you can close down the link. Thanks :)

[ROBYER1]

Can you also share your YoloV4 model? I'll check what is missing to make it work.

I found examples of Yolo v4/Yolo v4 tiny cfg/weights file using COCO data set - names file (if needed) at the popular AlexeyAB Darknet repo

Yolo v4 is being encouraged to be used for Yolo training so if any work can be done to implement it in Barracuda that would be fantastic, the structure should be more similar to v3, since v3 was a big departure from v2 as I understand.

Yolo-v4-tiny cfg pre-trained weights

Standard Yolo-v4 cfg pre-trained weights

I hope these files help