search

jimbobbennett / Xam.Plugins.OnDeviceCustomVision

Plugin to allow custom vision models to be used from mobile devices
MIT License
57 stars 13 forks source link

Question: object detection? #12

Closed nexxuno closed 4 years ago

nexxuno commented 4 years ago

I'm experimenting with object detection using a custom vision generated model with no luck so far. Are you thinking about implementing it in the plugin?

AmSmart commented 4 years ago

I built an app that used Object Detection recently. I don't think this project supports this right now. However, if internet upload isn't a problem, you can go with Custom Vision's API. That's the route I went

You can also use native library bindings for Tensorflow, CoreML and ONNX

It's not so complex, so you could make a PR to this repo actually 😉

nexxuno commented 4 years ago

It is supported and it works, it was bugged but they fixed it.

AmSmart commented 4 years ago

Oh! Cool, Awesome then

AmSmart commented 4 years ago

@nexxuno Hi, just inspected the repo, and no signs of Object Detection. Are you sure, can you point it out to me please?

nexxuno commented 4 years ago

Sorry, I tought I was on another repo. I had to implement everything in two flavours (ios, android) but I didn't have the time to pluginize it. It works good and you don't have to create the native library bindings since they are already in nuget. There are also some good tutorials online (I can't find the links now). I achieved 6 frames per second (acquisition + object detection) with tensorflow lite on a huawei mate 20 pro, which is decent I think.

The bug I was referring to was in how the CustomVision model was exported for tensorflow lite, which was borken for object detection but MS fixed it when I reported the problem.

AmSmart commented 4 years ago

Awesome! If your project isn't private, sharing it would make it easier to write a plugin for it. Or you can just crop out the needed code into a Github gist or something. I'd be more than willing to write a plugin for it and reduce development time for other developers. Thanks in advance.

nexxuno commented 4 years ago

It's private and in a very bad shape. I'm not spending more time on it in the near future. iOS part wasn't even finished. For the android part you have to use camera2 api, subscribe to the event which outputs frames, convert the frame to the size accepted by the tflite model, pass it to the model.

This is a great starting point: https://devblogs.microsoft.com/xamarin/image-classification-xamarin-android/ (even if it's about classification)

AmSmart commented 4 years ago

Oh! Right. Thanks for the hint. I'll see what I can do.