search

anilsathyan7 / Portrait-Segmentation

Real-time portrait segmentation for mobile devices
MIT License
638 stars 133 forks source link

About the android project #5

Closed RobinHan24 closed 3 years ago

RobinHan24 commented 4 years ago

Could you please provide the android project, thanks a lot.

boehm-e commented 4 years ago

Yes it would be really nice :)

anilsathyan7 commented 4 years ago

Actually i need to clean up the messy android code and i'am planning to include some additional optimizations. I'am also waiting for some other related github issues to be resolved in this regard. Once it's done i will upload the android project as well.

If you find any errors, bugs or new ideas, please do share. Also, checkout the other links to android projects, for segmentation.

boehm-e commented 4 years ago

Ok, thank you :) I checked all the android segmentation links i could find, but they are running with to much latency for my purpose. What i am looking is more body segmentation than just portrait, but i look forward to test your implementation ! Thank you for your work :1st_place_medal:

anilsathyan7 commented 4 years ago

@boehm-e i was actually working on the same stuff. Which data-set did you use? Supervisely? The problem i feel is that if you want to capture more complexity, say full body instead of simple portrait, you need a model with larger input size or more number of layers. Again there will be a trade-off between accuracy and latency. For full body, 128 input size is way too small; but i think 224 will be sufficient.Finally make sure you have a sufficiently large and good dataset that fully represents your use case or needs. Off-course, it would be very challenging job to get a fully generalized model with very high accuracy and low latency.I think it would be better idea to try a larger/complex(mobile friendly) model first and achieve desired accuracy.Afterwards, you can do all the engineering stuff for optimizations. Did you try deeplab and what is you target latency/fps?

boehm-e commented 4 years ago

@anilsathyan7 I am absolutly not a machine learning expert, so I just tried things other people have done. I tried deeplab implementation on android device https://github.com/tantara/JejuNet and https://github.com/joaquim-verges/DeeplabAndroid (demo)

The second project runs at about 2-3 fps on my honor 10. But it classifies all of deeplab classes, and as many people, i just would like to segment people (at higher speed like >15 fps) Do you know if it is possible to achieve that on android ? Thank you :)

anilsathyan7 commented 4 years ago

In the demo video that i've shown, the model actually runs at 10-15 fps in my oneplus3. You should get almost the same speed(may be better since you have a better GPU..) using the same model. In fact we tried them on Honor Play, which has almost the same spec's as Honor 10. If you have a better phone with chips like Sanpdragon 855+Adreno 640 or Kirin+Mali-G76 you can easily get 20 fps.[Still they can't beat apple's iphone ...😜]

I shall upload the app soon ...

boehm-e commented 4 years ago

Ok, and in iOS it is human segmentation, do you think it is because their model is well optimized, or it comes from computing power?

anilsathyan7 commented 4 years ago

According to the official paper, "In iOS devices, we benefit from larger cache sizes that result in reduced memory I/O latency, and hence,better performance than the OpenGL backend". tflite_speed

boehm-e commented 4 years ago

@anilsathyan7 Thank you for releasing the app :) Will try it as soon as possible :100:

boehm-e commented 4 years ago

It works pretty well :) I have 10-12 FPS on my Honor 10 (RIP my pocophone f1) Can I know what is the "munet_mnv3_wm10.tflite" file ? Thanks

anilsathyan7 commented 4 years ago

I tried new mobilenetv3 architecture instead of mobilenetv2 with a higher input size i.e 224...It's faster than corresponfing v2 version; but slightly less accurate. Again to make it even lighter i reduced number of filters and trained on slightly different dataset. But in the end it's test accuracy decreased ...

Tomyxtrejo commented 4 years ago

It is possible to have a pre-build apk to test performance? Project looks really good, I get interested in this when I see the TikTok real time segmentation... wow, that impressive

anilsathyan7 commented 4 years ago

@Tomyxtrejo You can use the tflite benchmark tool to benchmark the performance i.e speed of the model(alone) on android.Also you can create apk once you open the project with android studio(Build APK).Performance depends on your hardware and latest tflite gpu delegate (tensorflow); besides my current (unoptimized) code.If you a referring to real-world accuracy (or visual appeal), there seems to a lot of scope for improvement.I'am still trying to improve the performance in this regard !!!

By the way did you mean 'green screen effect' when you referred to real-time segmentation in tk-tok? Which filter is it?

Tomyxtrejo commented 4 years ago

@anilsathyan7 Ok I tryed, but the apk give me an error, I will try again. I don`t have Android Studio and I saw that wasn't that easy for a newby to setup the project, but will give a try.

Yes! the green screen filter in tiktok... For a instance i think that was some kind of processed in the cloud and displayed in my low gamma device (samsung j2 prime) but then I activate airplane mode and still working the same... Is incredible how can go to 15-20 fps at least and have a transparency edges that make it look more soft. And I found 0 info about that filter online.