j20001970 / GDMP

Godot plugin for MediaPipe framework and solutions
MIT License
66 stars 13 forks source link

Adding MediaPipeHolisticLandmarker to MediaPipe v0.10.13 #37

Open GeorgeS2019 opened 6 months ago

GeorgeS2019 commented 6 months ago

@j20001970 Is it feasible to upgrade to the latest MediaPipe v0.10.10 so the community could take advantage of the latest feature :

Holistic

https://chuoling.github.io/mediapipe/solutions/holistic.html

GeorgeS2019 commented 6 months ago

The latest is v 0.10.11

The ongoing discussion

GeorgeS2019 commented 6 months ago

@j20001970 Is there a need to coordinate upgrading to v0.10.11

Will that break existing version?

GeorgeS2019 commented 5 months ago

https://github.com/google/mediapipe/releases/tag/v0.10.13

Attempting v0.10.11 here. https://github.com/Malcolmnixon/GDMP/actions

The feedback to mediapipe community https://github.com/google/mediapipe/issues/5366

j20001970 commented 5 months ago

@j20001970 Is there a need to coordinate upgrading to v0.10.11

Will that break existing version?

As the issue you linked, currently there are problem in using preprocessor directives within function-like macro on Windows, bumping to mediapipe v0.10.11 will fail to build Windows library without a proper fix. Since the issue has already been opened on upstream mediapipe repo, I think there is no need to take actions further, let's wait for next release tag hopefully fix the issue.

In the meantime, try not to urging people too hard on fixing certain things, different projects have different goals and priorities in mind.

GeorgeS2019 commented 5 months ago

GDMP is now finally updated to the latest Mediapipe v0.10.13 and built successfully https://github.com/Malcolmnixon/GDMP/tree/test-mediapipe Great Job Malcolm

Malcolmnixon commented 5 months ago

I've submitted the preprocessor-fix PR to mediapipe (https://github.com/google/mediapipe/pull/5392) and once it's in we could update to a commit, or wait for v0.10.14 which would probably be about a month out considering the average time between releases.

GeorgeS2019 commented 5 months ago

https://github.com/j20001970/GDMP/pull/40#issuecomment-2103884722

GeorgeS2019 commented 5 months ago

Mediapipe

https://github.com/google/mediapipe/tree/master/mediapipe/tasks/cc/vision/holistic_landmarker

GDMP

holistic_landmarker.h holistic_landmarker.cc

Documentation

MediaPipeHolisticLandmarker

MediaPipe Holistic requires coordination between up to 8 models per frame

  • 1 pose detector,
  • 1 pose landmark model,
  • 3 re-crop models and
  • 3 keypoint models for hands and face.

https://chuoling.github.io/mediapipe/solutions/holistic.html https://github.com/google/mediapipe/blob/master/docs/solutions/holistic.md https://developers.google.com/mediapipe/api/solutions/js/tasks-vision.holisticlandmarker

References

https://github.com/j20001970/GDMP-demo/discussions/18

Malcolmnixon commented 5 months ago

Mediapipe currently don't have their official C++ API for Holistic Landmarker. A PR (https://github.com/google/mediapipe/pull/5063) has been started, although it does appear to have stalled for some number of months.

The release notes for 0.10.13 state "Make Holistic C++ graph public until we have a C++ API", so possibly we can implement the GDExtension classes as if the C++ API were present - inferring from the Holistic Landmarker PR and the APIs published for other languages (Java, Python, etc.). The implementation would hide the fact that it's using the C++ graph for now, and hopefully we can seamlessly switch over to the real API once it's published.

GeorgeS2019 commented 5 months ago

@Malcolmnixon Please help me here. Do we need to wait to have a working v0.10.13 dependent GDMP with the MediaPipeHolisticLandmarker ?

Malcolmnixon commented 5 months ago

Do we need to wait to have a working v0.10.13 dependent GDMP with the MediaPipeHolisticLandmarker

I'm not sure exactly what you're asking - GDMP is currently based on mediapipe v0.10.13 (with the necessary patches applied in the build process) and appears to work fine in my testing.

I'm suggesting the next step is to design the GDMP MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult objects and their associated methods in such a way that they can be implemented using the v0.10.13 Holistic C++ graph API, but also with the intent of switching to the proper Holistic C++ API once mediapipe releases it. That way anyone writing code based on the Godot API will hopefully not experience API breakage going forwards.

GeorgeS2019 commented 5 months ago

I'm suggesting the next step is to design the GDMP MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult objects and their associated methods in such a way that they can be implemented using the v0.10.13 Holistic C++ graph API, but also with the intent of switching to the proper Holistic C++ API once mediapipe releases it. That way anyone writing code based on the Godot API will hopefully not experience API breakage going forwards.

I am not C++

Are the c++ examples of v0.10.9 helpful to achieve the design and implementation for MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult?

How would you like others to help and coordinate?

Malcolmnixon commented 5 months ago

How would you like others to help and coordinate?

My plan (hopefully this weekend) is to familiarize myself with how GDMP maps the existing mediapipe APIs to GDExtension; and then take the unfinished Holistic Landmarker C++ API PR and apply the same mapping patterns to produce a proposed GDMP GDExtension API for the holistic landmarker with the same architectural/design choices as the existing tasks.

If that looks good then I'll try to implement that proposed GDExtension API, but using the unfinished Holistic Landmarker C++ API PR as a guide to for how to map the calls to the underlying graph API.

It would be simpler to just wait for mediapipe to release their C++ API; however that PR has stalled with no changes for 4 months, so I don't see any alternative but to work around the problem. To misquote Voltaire "Don't make great the enemy of good"!

GeorgeS2019 commented 5 months ago

Don't make great the enemy of good"! We shall discuss further on this :-)

KIV

https://github.com/google-ai-edge/mediapipe/releases/tag/v0.10.14

Is there something needed to be there but not yet?

GeorgeS2019 commented 3 months ago

MediaPipe v0.10.14 attempted

@Malcolmnixon @j20001970 https://github.com/homuler/MediaPipeUnityPlugin/pull/1200