How hard would it be to port to mobile? (iOS/Android)

[kibiz0r]

I'm looking for hand recognition that works on a mobile device. The videos from OpenPose are the best results that I've seen.

Two questions:

1. When will the hand recognition stuff be available? (Sick of that question yet? 😜 )
2. What are the obstacles to making this work on mobile?

I saw issue #27, but I was also looking at Tensorflow as a possibility.

I expect that reducing the size of the model will be necessary for practical use on mobile since wireless download and fast startup are important features, and it's okay to lose some quality for the sake of performance.

I'm pretty new to this stuff, so excuse me if I'm totally wrong. Any input at all would be greatly appreciated.

[mootpointer]

I'm trying to convert the .caffemodel to CoreML to see if I can run it on iOS 11. I will report more as I progress.

[kibiz0r]

Sweet. That's great to hear! Good luck, sir.

Any plans for Android?

[pennomi]

@mootpointer Did you have any luck converting the model?

[melito]

I've managed to produce a model that I can import into xcode and make predictions with.

I have no clue what I'm doing so I'm not sure how correct what I've put together is.

Here's a gist explaining: https://gist.github.com/melito/da9c408d8703b326226255d30e32d3c5

[pennomi]

@melito Thanks, I'll take a look.

[superying]

@kibiz0r Maybe Caffe2 is exactly what you are looking for. Try to implement openpose in caffe2.

[gineshidalgo99]

if somebody is able to implement it in caffe2, make a pull request and we'll try to add it and maintain it! 👍

[LoneWolfDog]

@mootpointer , Any progess on use openpose on IOS ?

[s1ddok]

@ChenYingfeng I successfully converted model and used it in iOS application. However idk what to do with its output. It gives me 1x1x22x46x46 array and I can't find docs on how to interpet this data

[xhzzhang]

I need find build solution in Android, is anybody have some advice? thanks.

[superying]

@s1ddok I get a similar 1x57x46x46 array result in Caffe2. And working on post process now. I suggest you work on post process(details in PoseExtractorCaffe::forwardPass) with your array result.

[boaerosuke]

I finally managed to solve handwritten mnist digit recognition with Tensorflow/Keras in iOS. Have a look if you want. This might definitely be portable to Android as well.

https://github.com/boaerosuke/digitrecognition_ios

[tangert]

@s1ddok @superying would you guys mind sharing your code? I'm trying to use the model in an app as well!

[nigellima]

@s1ddok @superying would you guys give us some tips on how to implement it with caffe 2 and run on a mobile device? I really to to try run this on android

[otmb]

Hi. @melito + image_scale

coreml_model = coremltools.converters.caffe.convert((caffe_model, proto_file)
, image_input_names='image'
, image_scale=1/255.
)

Sample Script at https://gist.github.com/otmb/7b2e1caf3330b97c82dc217af5844ad5

[snowzurfer]

To those who have tried running the model, even if you cannot understand its output, did you benchmark it? How long does inference of a frame take?

[s1ddok]

@snowzurfer very long, not even close to real-time on iPhone 7+ (over a second)

[snowzurfer]

@s1ddok I see; did you try to reduce the size of the input features like they say in the FAQs to get better performance?

[MorningMyFriend]

@superying hello, I have run openpose model with caffe2 python API on PC, it is slower than the original caffe. But i encounter problem while loading init_graph on Android. I wonder if you run the openpose model on Android or PC? How about the speed?

[otmb]

Hi. We released a swift-code to extract bones. But model processing is slow and not practical. https://github.com/infocom-tpo/SwiftOpenPose

Also, I tried transplanting about mobilenet of tf-openpose but did not work well with coreml. https://github.com/infocom-tpo/tf-openpose/tree/master/convert

Thank.

[Luttik]

If this is doable I want to take a look at it. (Don't have much time) did people start on the project already

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[MetaDev]

Has anybody made any progress on this?

[abdul-mateen-chughtai]

Has anyone got OpenPose running on Android?

[ghdestiny]

Has anyone got OpenPose running on Android?

[purbanow]

+1

[hassanilyas]

Any update for OpenPose running on iOS/Android?

[liyuan116]

Has anybody made any progress on this?

[cperbost]

Still no one?

[ljmkimqx]

+1

[kunall17]

+1

[otmb]

Hi. It's been 4 years since the previous post. Is anyone interested in working with iOS? On devices some time ago, the inference process was very slow. It works very fast on the latest iPhones.

Device	Release date	CPU	Around 1 frame
iPhone6s	2015	A9	2.2 sec
iPhone SE2	2020	A13	0.16 sec
iPhone12	2020	A14	0.07 sec

This is my recent job. It was created by porting the OpenVINO code, except for the model and PCM/PAF mapping information.

Sample Code https://github.com/otmb/SwiftOpenPose-body25/

Caffe to Coreml Convert https://github.com/otmb/SwiftOpenPose-body25/tree/main/convert

Thank.