Open DehTop opened 6 months ago
attaching here a couple of examples, also see https://github.com/google/mediapipe/issues/3994 for more (same issue)
wondering if anyone has a workaround/solution for this!
many thanks
Thanks for raising this. This looks like an issue with the model, but unfortunately we currently do not have plans to update our models.
Hello @schmidt-sebastian! I've been digging a bit more on this topic and wanted to share some interesting findings
At this point, I suspect that there could be a bug in how this internal rotation/alignment is performed when the back of the hand is shown to the model! In my mind, this is the only explanation for points 4) and 5).
Wondering if there is a rather easy fix at inference time for this issue, instead of having to re-train the model.
many thanks for your work, hope this is useful and can ring a bell in someone's mind for a quick inference fix!
Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
None
OS Platform and Distribution
Linux Ubuntu 20
MediaPipe Tasks SDK version
0.10.9
Task name (e.g. Image classification, Gesture recognition etc.)
Hand landmark detection
Programming Language and version (e.g. C++, Python, Java)
Python
Describe the actual behavior
In mediapipe v0.10.9, when detecting and visualizing the back of the hand, the 3d landmarks of the palm happen to collapse, especially the finger MCPs, producing weird and unusable results. This happens consistently across different lighting, poses, hands.
Describe the expected behaviour
The 3d landmarks should be consistent even when the back of the hand is shown. They should certainly not collapse, at least..
Standalone code/steps you may have used to try to get what you need
Other info / Complete Logs