search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
26.97k stars 5.1k forks source link

Custom object keypoint tracking #5165

Open SpirikleOfficial opened 7 months ago

SpirikleOfficial commented 7 months ago

MediaPipe Solution (you are using)

none

Programming language

Android java

Are you willing to contribute it

No

Describe the feature and the current behaviour/state

No feature for custom object keypoint tracking. Current feature offering includes keypoint tracking but limited to hand only. You can ask custom dataset which a user can provide, hoping that the user gets the required keypoint tracking model from your mediapipe studio. This will be a huge leap for the mediapipe studio and on-device ML.

Will this change the current API? How?

No response

Who will benefit with this feature?

Anyone who wants object keypoint tracking model.

Please specify the use cases for this feature

Can use to specify the corners of a document which can be used to later crop the document. If can find keypoints of any object, it will get easy to crop that object.

Any Other info

No response

ayushgdev commented 7 months ago

Hello @SpirikleOfficial Thanks for filing the issue in MediaPipe. Would you please elaborate the feature requirement, possibly with some example which you would like to give to MediaPipe, a sample output that you expect, and the internal processing that should automatically be handled by MediaPipe? Further, we do support keypoint tracking in HandLandmarker plus other solutions like HolisticLandmarker, PoseLandmarker and FaceLandmarker as well. Tracking is invoked after detection so that MediaPipe doesnt consume CPU/GPU resource on your edge device by runnning detection on every frame.

SpirikleOfficial commented 7 months ago

There can be multiple examples for custom keypoint detection. What you guys offer is limited and rigid options. No way for a developer to use their custom keypoint dataset and get the required model from your MediaPipe Studio. One example can be of detecting corners of a document, for cropping easily, which I mentioned earlier. I do not understand what you couldn't infer from the first post above? Everything I have mentioned clearly. Still, I'll be happy to elaborate again if required.