Pose Landmark Detection Jittering - Python Solution

[mcdonasd1212]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Ubuntu 20.04

MediaPipe Tasks SDK version

0.10.3

Task name (e.g. Image classification, Gesture recognition etc.)

Pose Landmark Detection

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I have switched over from the legacy mediapipe library to the new mediapipe solutions. The landmarks are jittering more than expected when I use the VIDEO runningMode on GPU or CPU with any of the pose_landmarker tasks.

Describe the expected behaviour

The legacy mediapipe pose estimation detection offered a smoothing parameter (smoothLandmarks) to reduce the jittering, which worked quite well. I have not been able to find this option in the new mediapipe solutions library.

Standalone code/steps you may have used to try to get what you need

The jittering can be observed on the official mediapipe solutions demo page:

https://mediapipe-studio.webapps.google.com/demo/pose_landmarker

Other info / Complete Logs

See issue #4507 for same issue with discussion regarding javascript implementation.

[kuaashish]

Hello @lu-wang-g,

Could you please look into this issue. Thank you

[mcdonasd1212]

@lu-wang-g @kuaashish

Really need some advice regarding this issue. Thanks!

[mcdonasd1212]

@schmidt-sebastian any recommendations?

[schmidt-sebastian]

We will be working on this shortly.

[mcdonasd1212]

We will be working on this shortly.

Is shortly by the end of the year? Trying to plan our development.

[schmidt-sebastian]

Yes.

[kuaashish]

@mcdonasd1212,

We are pleased to announce the release of the latest version of MediaPipe, version 0.10.7, which addresses the jittering issue observed in the Pose Landmarker.

This issue has been documented in the release notes under "Fixed Pose Landmarker jittering issue." We kindly request you to build using this updated version and inform us of any persisting issues from your perspective. Thank you

[mcdonasd1212]

@kuaashish and @schmidt-sebastian Thank you for the update. Using the same code I used with version 10.5 now that I have used pip to install 10.7 I am getting the following error

W0000 00:00:1697051228.935013  124391 one_euro_filter.cc:31] New timestamp is equal or less than the last one.

Here is a python code snippet

frame_timestamp_ms = mp.Timestamp.from_seconds(time()).value
mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)

results = pose.detect_for_video(mp_image, frame_timestamp_ms)

Help!

[schmidt-sebastian]

Please make sure that you use the time of the video frame (rather than the current time). If you cannot do that, use running mode "image".

[mcdonasd1212]

Please make sure that you use the time of the video frame (rather than the current time). If you cannot do that, use running mode "image".

@schmidt-sebastian the change took care of the error. Compared to the legacy version of MediaPipe, I am still seeing a lot of jittering. Is there a way I can verify that my solution is taking advantage of the fix that was deployed?

[kuaashish]

Hi @mcdonasd1212,

Did you try running MediaPipe 0.10.7 in Python? It seems to work when use the runningMode set toVIDEO and the detector.detect_for_video() function. Please let us know if you are still experiencing jitter. Thank you

[msabers]

Hi @mcdonasd1212,

Did you try running MediaPipe 0.10.7 in Python? It seems to work when use the runningMode set toVIDEO and the detector.detect_for_video() function. Please let us know if you are still experiencing jitter. Thank you

Hi, I am not a professional programmer, I used the code below with the MediaPipe version 10.1 but it was very jittery with some videos: mp_draw = mp.solutions.drawing_utils pose = mp_pose.Pose( min_detection_confidence=0.7, min_tracking_confidence=0.9, model_complexity=2, smooth_landmarks=True, static_image_mode=True, ) results = pose.process(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))

After upgrading MediaPipe to the latest version I changed the results part as below but it doesn't show anything and shows "no pose available" error: results = pose.detect_for_video(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))

I really need to increase the accuracy of my detection and I would appreciate it if you let me know why in some videos it is jittery.