search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
26.85k stars 5.09k forks source link

Flutter Package? #3922

Open charlieforward9 opened 1 year ago

charlieforward9 commented 1 year ago

Hello, just learned about MediaPipe today! It's amazing how many developer resources are open-source. It's also crazy to realize that Google has empowered so much of it. With that being said, I was surprised to see the MediaPipe team has not created a package for Flutter developers since both were founded by Google and the MediaPipe is marketed as cross-platform

System information

Will this change the current API?

No. Creating a Flutter package will extend and merge the iOS/Android API such that the new, extended API can be used natively in the Dart language. Nothing about the current API will need to be changed as far as I am concerned.

Who will benefit with this feature?

kuaashish commented 1 year ago

Hi @charlieforward9, Thanks for raising this feature request. This looks like an interesting request. We will discuss this feature internally and prioritize it among others.

szotp-lc commented 1 year ago

I have integrated mediapipe into my flutter app but it was really cumbersome. It would be great at least have C api that can be called directly from dart, instead of channel communication separate for each platform.

There is https://github.com/homuler/MediaPipeUnityPlugin/tree/master/mediapipe_api, originally created for Unity but could be repurposed. The problem is - it's not pure C, so generating ffi bindings is not so straightforward, and it is not official.

charlieforward9 commented 1 year ago

@szotp-lc

It would be great at least have C api that can be called directly from dart, instead of channel communication separate for each platform.

If it were done this way, would it work on all the mobile and desktop platforms that Flutter supports? If so, then the only separate implementation would be the web? Asking to learn. Thanks

szotp-lc commented 1 year ago

@charlieforward9 Correct.

schmidt-sebastian commented 1 year ago

Hi! We are having some discussion about this, but cannot provide a timeline at this point. We certainly recognize that better Flutter support is very important to large part of our community. We will provide updates if and when we can.

charlieforward9 commented 1 year ago

@szotp-lc do you mind emailing me at charlesrichardsonusa@gmail.com, I would appreciate hearing more about how you integrated into your Flutter app (unless you didn't mind keeping the conversation in this thread). Thank you!

szotp-lc commented 1 year ago

Honestly I would recommend waiting for official support or using a different library (like ML Kit). It has been a painful process.

charlieforward9 commented 1 year ago

Honestly I would recommend waiting for official support or using a different library (like ML Kit).

Official Support > .tflite embedding > MLKit MethodChannel

I built a PoseDetection MethodChannel for iOS using MLKit to compare it with my existing pose_detection_full.tflite file embedding using flutter-tflite. The MethodChannel inference delivered slower than the .tflite file embedding with my implementation.

I am sharing this to show findings, save other people time, and keep this relevant.

Goal

Actual (Embedded .tflite)

https://github.com/google/mediapipe/assets/62311337/26d8be7b-6db4-423e-b54e-04dc3144cc29

Actual (iOS MLKit Method Channel)

https://github.com/google/mediapipe/assets/62311337/b6f09123-1983-497d-8b21-65572874ee2e

My implementation uses Dart Isolates to keep the model processing off of the main thread.

charlieforward9 commented 1 year ago

Thanks to Google and their Mediapipe team, we are now able to witness development of flutter-mediapipe. It's great to follow along and see how Google would build a package from the ground up.

I have become a watcher of the repository to get a firsthand sight of the progress and changes, so if anyone else lands here and is interested in the progress of the package. I recommend doing the same.

rukinish commented 6 months ago

so as of now is it possible to have a mobile application that is developed using flutter that has mediapipe ?

charlieforward9 commented 6 months ago

Yes, there is just a lot of manual setup. I recommend using this sample repo to help get started with it.

I didn't need Android config. If you manage to get it up and running with all updated dependencies, feel free to create a PR to Open Source it.. I am waiting for the original creator @Jaeheee to merge it, so we can keep my fork current till it merges.

Happy Coding!

charlieforward9 commented 4 months ago

As a humbled developer who requested this over a year ago. I have subscribed to the construction and have been nothing short of impressed.

I am particularly excited to see Pose Detection begin its development as an included feature at https://github.com/google/flutter-mediapipe/issues/51. It was the Mediapipe solution I was having trouble running performantly that prompted me to open this issue.

Thank you Google AI

iarshguleria commented 3 months ago

@charlieforward9 I have been implementing mediapipe-pose using JS but now I want my application to built for multiple platform on one codebase so I have been trying to migrate to flutter, just wanted to ask whether using platform channel solve that problem as of now or is there some alternate

charlieforward9 commented 3 months ago

You can see the performance of Method Channels above. @iarshguleria

I am really hoping for the official repo to perform closer to the "goal".

I am getting started on the vision package for the team.

This is my first large scale contribution, so I cannot provide a timeline at the time. However, I will continue to update as I proceed.

https://github.com/google/flutter-mediapipe/discussions/62

iarshguleria commented 3 months ago

Method Channel doesn't seem to have that great performance as compared to the native code