Yolo v4 support?

[ROBYER1]

I was working with Yolo v2 until I saw 1.2.0 added v3 support, will there be Yolo v4 support added soon also?

[FlorentGuinier]

Hey, We are interrested into supporting Yolo v4 indeed, however not ETA atm.

[Ademord]

how could I get yolov3 running? everytime i import it i get some error

[FlorentGuinier]

conversation is happening here https://forum.unity.com/threads/yolo-tiny-v4-support.1016356/

[ROBYER1]

Any update on Yolo v4 support also?

[FlorentGuinier]

Not at the moment as other things are being prioritized. Please let us know if yolov4 support is mandatory for your use case however.

[Ademord]

Not at the moment as other things are being prioritized. Please let us know if yolov4 support is mandatory for your use case however.

@FlorentGuinier I ran the yolov2 thats available here, and i get okay results.

however, this yolov2 variant i am using is limited to 20 classes, and also it doesnt recognize a lot the 3d models i imported, even if they are looking realistic to me :/

so i tried to use the yolov3, as suggested in the link you provided. they pointed me to this repo: https://github.com/derenlei/Unity_Detection2AR since @ROBYER1 is on holidays.

yolov3 gives me no results at all :/. for some context, i am using the yolov2 for Barracuda from this repo (thanks @keijiro).

and adapted the config from yolov2 to yolov3 as follows

        public const int CellsInRow = 13;    // yolov2 = 13
        public const int AnchorCount = 5;    // yolov2 = 5
        public const int ClassCount = 20;       // yolov2 = 20
        /*
        public const int CellsInRow = 26;       // yolov3 = 26
        public const int AnchorCount = 3;       // yolov3 = 3
        public const int ClassCount = 80;       // yolov3 = 80
        */
        public const int InputSize = ImageSize * ImageSize * 3;
        public const int TotalCells = CellsInRow * CellsInRow;
        public const int OutputPerCell = AnchorCount * (5 + ClassCount);
        public const int MaxDetection = TotalCells * AnchorCount;

is there anything i could do to improve from the current point i am at?

[FlorentGuinier]

Hi @Ademord weird that yolov3 provide no result at all. I think the first thing i would try is to split the problem to identify where the bug is:

• run the model in python with the native lib it was trained with + an example of the training set.
• move to onnx (maybe try to run with onnxruntime)
• move to Barracuda (still with training set example as a texture).

Then once the pipeline is proven to work:

• Barracuda inference with in engine feed

Hope it helps! Florent

[Ademord]

@FlorentGuinier thanks for your quick reply. before I jump into debugging the whole model, do you guys have any official yolov3 usage docs ? not that I have to continue looking at other random repos (and reverse engineer them), to see how they used barracuda?

edit: i have also started a ticket with the unity perception module since it gave me some errors from the simulation package. so i am trying a lot of different points of view :P

[FlorentGuinier]

Unfortunatly we don't, we support many different models so that would not scale for the team if we did. Best is probably to look at repos from the community indeed!

[Ademord]

@FlorentGuinier thanks again for your reply. Do you know of any way I could use Pytorch3D in Unity? Either export data from the engine outside or use Pytorch3D inside ? I have no clue how could i send data to a service or...i am sure there must be a way.

Edit: do you think Barracuda would support a pytorch3d model (onnx?)

Edit2: for that matter, i am looking at pytorch3d because they allow calculating a chamfer loss for two meshes quickly.

[amirebrahimi]

@Ademord - Please create a new issue if need be instead of posting comments on closed issues as it is hard for us to track or if it is more general discussion I'd suggest bringing it to our forum. Also, original authors of issues may get notified of these comments on old issues, which may not be relevant to them. It's better to create a new issue referring to an old one with the "#< issuenumber >" macro that github supports.

Re: your question - we are a small team and don't have the resources to check whether Barracuda supports X for all the X that exist out there. Instead, we try our best to provide enough information for you to be able to answer that question.

Without knowing what pytorch3d encompasses (or any X for that matter) I can't answer your question directly, but the way you could answer this would be:

1. Does it support exporting to ONNX file format?
2. Does it run in onnxruntime? If it doesn't, then it likely won't run in Barracuda.
3. What operators does it require? Here is our list of supported operators
4. Does it require any pre-processing that normally exists in Python? If so, then you'll likely need to convert that into C# code.
5. Does it require any post-processing that normally exists in Python? If so, then you'll likely need to convert that into C# code.

I hope that this is helpful.