derenlei / Unity_Detection2AR

Localize 2D image object detection in 3D Scene with Yolo in Unity Barracuda and ARFoundation.
https://derenlei.medium.com/object-detection-with-localization-using-unity-barracuda-and-arfoundation-794b4eff02f8
MIT License
223 stars 61 forks source link

Custom Trained YOLOv3 Tiny Model not detecting anything #29

Open DaphiFluffi opened 3 years ago

DaphiFluffi commented 3 years ago

Hi again,

I wanted to open up a new issue about this. I trained a custom model using Darknet. It is trained to detect only one class, a pillbox. And when tested in Darknet, it did. I trained the model for 4000 iterations. Then I converted it to Tensorflow, froze the graph and converted it to ONNX successfully. Then I was able to import it into Unity. I adjusted the anchor values to my own and filled out the Detector in the Inspector. But, unfortunately, nothing is detected when I build the app on Android. But there are also no errors. What could the mistake be? Could it have something to do with the input shape?

Any help is very apprechiated :) import detector

Ademord commented 3 years ago

Same for me I dont detect anything when I imported the yolov3 model linked in this repo.

DaphiFluffi commented 3 years ago

the default one works for me, but not my custom one

Ademord commented 3 years ago

did you integrate the detector into a new code? if you could help me understand how do i reuse it in another class? (inputs and format, outputs and format?)

DaphiFluffi commented 3 years ago

no not really. I am just using the ARAnchorManager's Prefab to instantiate my own objects on top of the detected object. I only reverse engineered that part.

Ademord commented 3 years ago

ah so you are doing some variant of the face-filter scenario... i see. i am trying to understand how to use any kind of yolo (onnx) and pass it the camera input (render texture), and then output the class and bbox. i got the code from https://github.com/keijiro/TinyYOLOv2Barracuda working but now I need to upgrade it to yolov3 and idk how to (yet).

DaphiFluffi commented 3 years ago

I don't do anything with face decetion. And if everything you want is to detect an object from your phone's current camera feed, this is exactly what this repository does. And it works with Yolo v3 for me as well. Do you have Errors? Did you export it to Android successfully? Which Objects did you test with?

Ademord commented 3 years ago

i am trying to integrate yolov3 using barracuda into a game's camera feed, not using a phone or a webcam. most repos out there using barracuda (not many in general), are using phone or PC cameras...

DaphiFluffi commented 3 years ago

Ok, i don't know much about that. But you should read into how the AR Camera in the AR Session Origin captures the Phone Camera's Feed. I think it would be possible to change it out quite easily.

Kanyade commented 3 years ago

Hi @DaphiFluffi @derenlei @ROBYER1 @artoonie @BeiieB Is there any news on this? Have you solved it? I am in the exact same boat. The model works elsewhere, just not within Unity. image

The main issue I think is the fact that the original yolo v3 model in the repo has output_l(conv16) and output_m(conv23) values and our custom model has one output. image

This is the error I could extract from adb debugger. image Any tips would be appreciated.

Kanyade commented 3 years ago

I managed to have two outputs by changing the tensorflow freeze graph function output node names to yolov3-tiny/convolutional10/BiasAdd,yolov3-tiny/convolutional13/BiasAdd and setting this in tf2onnx as well. I did it because at the end of the DW2TF conversion script it shows the output layers:

image

I did the conversion by your guide @DaphiFluffi https://github.com/DaphiFluffi/Yolov3-To-Onnx-Unity that's why we have really similar results.

And also a weird thing to note with the onnx formats is that the outputs' shape changes to -1 -1 -1 -1 but in Netron it shows like this:

image

Could it be because of the unknown value?(First in outputs) Any idea how to solve it?

Ademord commented 3 years ago

I managed to have two outputs by changing the tensorflow freeze graph function output node names to yolov3-tiny/convolutional10/BiasAdd,yolov3-tiny/convolutional13/BiasAdd and setting this in tf2onnx as well. I did it because at the end of the DW2TF conversion script it shows the output layers:

image

I did the conversion by your guide @DaphiFluffi https://github.com/DaphiFluffi/Yolov3-To-Onnx-Unity that's why we have really similar results.

And also a weird thing to note with the onnx formats is that the outputs' shape changes to -1 -1 -1 -1 but in Netron it shows like this:

image

Could it be because of the unknown value?(First in outputs) Any idea how to solve it?

Hey, just to clarify which yolov3 model did you use and what command from the source you mentioned? If you could share the command you used to replicate? And in Unity to load the model did you change the parsing of the outputs? Would you have a very small project to make replicate it at best?

Hi @michalek-marcin ,

Did you freeze the tf graph to convert model parameters into constants? You may need tf weights (.ckpt, you can get it here: DW2TF) and the tf graph (.pb). Somthing like this: python3 -m tensorflow.python.tools.freeze_graph --input_graph=./checkpoint_dir/yolov2-tiny.pb --input_binary=true --output_node_names=yolov2-tiny/convolutional9/BiasAdd --input_checkpoint=./checkpoint_dir/yolov2-tiny.ckpt --output_graph=./checkpoint_dir/yolov2-tiny-freeze.pb

I then convert the frozen graph to onnx through: python3 -m tf2onnx.convert --graphdef ./yolov2-tiny-freeze.pb --output yolov2-tiny.onnx --inputs yolov2-tiny/net1:0 --outputs yolov2-tiny/convolutional9/BiasAdd:0

Make sure you use the correct graph inputs and outputs.

Kanyade commented 3 years ago

@Ademord it was a custom model, not the one in the repo. I followed @DaphiFluffi 's guide which I referenced with these modifications to have two outputs:

image

image

But I still don't know how to get it to work inside Unity, maybe I'll try to test the onnx model itself outside Unity to see if the problem is with the mode itself

Ademord commented 3 years ago

@Kanyade thank you so much! and to integrate this into the render did you change any parts of the code? im struggling with this integration for a while now

Kanyade commented 3 years ago

@Ademord

I did not change anything, if you download the original repo then it should work fine with the correct Asset versions with the original onnx model.

I completely aborted the custom model I did while writing the previous comments as the output shape was invalid and the conversions must have gone wrong.

Here I solved the custom model problem, I'll post it so others can benefit from it as well:

https://gist.github.com/Kanyade/c8ee3b7eca6f6e470e418b018ea02e83

Special thanks to @jkjung-avt for this repo: https://github.com/jkjung-avt/tensorrt_demos#yolov3 and to @prratadiya for this repo: https://github.com/prratadiya/tensorrt-installation-colab

It works in the project now, it is the same logic behind the conversion as how the default model was converted -- with TensorRT --.

The only problem is that now the last shape of the output values is reduced compared to the default onnx model in the demo and that the detection boxes are bigger than my mobile screen. I think there is a correlation but I did not dive deep into it yet.

image image

The yolov3-tiny-416 is the original from the repo and the yolov3-tiny is mine

Ademord commented 3 years ago

I am using the yolo model and on linux my build is giving me an error I have document on this issue https://forum.unity.com/threads/urgent-shader-loading-problem.1194259/ Could anyone help me here?