How to reduce the jittering of face landmarks?

[Mad-Thanos]

Hi Mediapipe cool guys, I read the blog about the Face Mesh feature here:https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html?m=1

The realtime mesh is so attractive to me, it's incredibly fast and dense. But when I tried the Mediapipe Face Mesh prebuilt demo apk(downloaded from this page:https://google.github.io/mediapipe/solutions/face_mesh ). I found that everything is good but the dense landmarks are jittering(i.e. shaking) even when I keep the phone and my head static. Is there any option or method to reduce or eliminate the jittering? What could be the cause of the landmark jittering?

2 Android phones were tested for this demo apk, one is powered by Snapdragon 845 and another is Kirin 980.

Thanks!

[fanzhanggoogle]

Hi, we do expect some jittering as the model released in MediaPipe is extremely light (lighter than the model announced in the blogpost). When deploying the model in production, we apply a low pass filter to get temporally stable prediction . You could give it a try.

[Mad-Thanos]

hi, @fanzhanggoogle Thank you!

Is the high definition model publicly available?

I'm not sure whether it is enough to use a simple filter, such as a sliding window average filter, is it enough to reduce the mesh point noise by averaging the point coordinates on the most recent 3 or more meshes(in a sliding window) ?

[Mad-Thanos]

I will try the sliding window filter first, and then some more complex filter if possible.

[mgyong]

@Mad-Thanos The high definition model is not publicly available and we do not have plans to release. We plan to provide a new update of the Face mesh model in July/Aug if you have a simple filter solution. Please contribute back to the community. Thanks

[timkabot]

@Mad-Thanos Did you succeed?

[hananbeer]

I tried running lerp() (linear interpolation) for 10 time steps between each model prediction. not perfect, so I also tried updating a keypoint only if the distance of the new keypoint is above certain threshold. not sure which method is better.

the problem is I only work in 2D points, I completely ignore the Z position (depth), but the depth seem to be the most jittery. sadly even still images provide jittery output.

[Mad-Thanos]

@Mad-Thanos Did you succeed?

Hi @timkabot and @hananbeer , I didn't work on this topic anymore. But I think google mediapipe team had already implied some clues about this topic in the original paper(pls googling mediapipe facemesh paper). They said they used the one-euro filter to smooth out the jittering of each coordinate(x,y,z) of the 3D points. There are already some one-euro filter implementations on github, for example:https://github.com/jaantollander/OneEuroFilter

Looking forward to you guys success.

[delebash]

@mgyong Why would you all not release the high definition for all models? I am working on using the hands model for 3D motion capture animating a 3D character but because of the jitter it is not suitable to use. That would be wonderful to be able to use a model that would not produce jitter. Please consider releasing. Thanks!

[amazingyyc]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

[Dhruv2911]

@amazingyyc I also have the same jittering problem, but i dont know how to implement the filter in javascript, Do you know where i can find a tutorial or the code to implement it on webcam in javascript!!

[amazingyyc]

@amazingyyc I also have the same jittering problem, but i dont know how to implement the filter in javascript, Do you know where i can find a tutorial or the code to implement it on webcam in javascript!!

Sorry I also donot know How to implement KalmanFIlter in JavaScript, I use opencv to implement this is the OpenCV tutorial:https://docs.opencv.org/master/dd/d6a/classcv_1_1KalmanFilter.html

[fahadhaq]

Where do I need to add the filter to? I am playing around with the android example currently and trying to figure this out. Any help would be greatly appreciated.

[delebash]

MediaPipe Team, first thanks for the great product. I am just checking back here and after trying the latest update v0.8.4 in javascript the jitter issue still remains. Because of this it is impossible to use for 3D motion capture to animate a 3D character. Is there a status update on fixing this issue? Adding a filter currently does not seem to fix the problem well enough as I have seen other projects attempt at smoothing the jitter but none well enough to use. I know this can be done as companies like wrnch ai and radical motion both can capture from a webcam and animate a 3D character without the jitter. Looking forward to future updates on this. Thanks again for your hard work!!!

Dan

[emo10001]

Like everyone else, I too am experiencing the jittering. Was looking to try to implement this for Facial Mocap, but my fears is this will be unusable.

I have no idea how to implement KalmanFilter as I'm just a hacker at best :) Has anyone successfully implemented this in OpenCV python?

Seems disheartening that hi-def models won't be released.

[brucechou1983]

In fact, users don't need to implement the filter themselves. There is a LandmarksSmoothingCalculator defined in build target //mediapipe/calculators/util:landmarks_smoothing_calculator. Face Effect demo app actually use this calculator to reduce jittering of face mesh module. If you check both face effect and face mesh demo apps carefully, you can see obvious difference in landmark stability.

From a model perspective, if you found the open source model doesn't fit your datasets/requirements, you could just fine tune it using your datasets/augmentations/loss functions. The model structure and weights are all available so porting it to your training pipeline should be straightforward.

[delebash]

How about hands and pose do those have this same ability. Also some people do not know how to do what you are suggesting so could you elaborate on the steps? I don't really know how to create my own model/datasets/loss functions. I just want to use the app features. Also does it reduce the jitter enough for use in motion capture for 3D animation?

[brucechou1983]

How about hands and pose do those have this same ability.

I think so

[showbit01]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

[amazingyyc]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

[showbit01]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

[amazingyyc]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

This is my app:https://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063 that use the Kalman filter to reduce the jitter. You can try it to make sure whether the method is usedul for you.

[tangerine1202]

For anyone who wants to learn about Kalman Filter, this guy explained it clearly without going too much into mathematical detail.

https://nbviewer.jupyter.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/00-Preface.ipynb

[showbit01]

@tangerine1202 Hey thanks for this.Would you please share some idea about how to get the accurate Yaw, pitch,roll from mediapipe facemesh.

[tangerine1202]

@showbit01 I think it might be difficult to deal with yaw, pitch, roll, since it cannot be observed directly and it's also difficult for people to tell if the values make sense. Try to get the accureate x, y, z and dx, dy, dz, then inference yaw, pitch, roll from it might be better.

I accidently saw another filter called "one euro filter", which is much simpler than Kalman Filter. It may be worth a try.

One Euro Filter explaination and python impl: https://jaantollander.com/post/noise-filtering-using-one-euro-filter/

[showbit01]

@tangerine1202 mediapipe facemesh gives 3d screen landmarks x,y,z where z is the depth from virtual plane assumed ,if we get the real world coordinate of face in front of screen then with world 3d and image 2d, rotation matrix will be calculated from then yaw pitch roll euler angles obtained. Finding world coordinate will solve the same.

So is it for same purpose the link you provided?

[ecartman2208]

In fact, users don't need to implement the filter themselves. There is a LandmarksSmoothingCalculator defined in build target //mediapipe/calculators/util:landmarks_smoothing_calculator. Face Effect demo app actually use this calculator to reduce jittering of face mesh module. If you check both face effect and face mesh demo apps carefully, you can see obvious difference in landmark stability.

From a model perspective, if you found the open source model doesn't fit your datasets/requirements, you could just fine tune it using your datasets/augmentations/loss functions. The model structure and weights are all available so porting it to your training pipeline should be straightforward.

I have seen that they have the OneEuroFilter implemented in the file. How can I integrate it in the python solution for facemesh that is available?

[PeterTF656]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

This is my app:https://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063 that use the Kalman filter to reduce the jitter. You can try it to make sure whether the method is usedul for you.

@amazingyyc

Dear yyc, I checked your app on ios platform. Did you build that app with flutter? It works fine from my perspective.

Peter

[amazingyyc]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

This is my app:https://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063 that use the Kalman filter to reduce the jitter. You can try it to make sure whether the method is usedul for you.

@amazingyyc

Dear yyc, I checked your app on ios platform. Did you build that app with flutter? It works fine from my perspective.

Peter

No it's not it build on swift+OC+CPP

[PeterTF656]

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

This is my app:https://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063 that use the Kalman filter to reduce the jitter. You can try it to make sure whether the method is usedul for you.

@amazingyyc Dear yyc, I checked your app on ios platform. Did you build that app with flutter? It works fine from my perspective. Peter

No it's not it build on swift+OC+CPP

Cool... I asked because Flutter probably can't tell the sampling rate of camera image stream, which is essential for all kinds of filters. While in native, I guess it would be much easier to find the sampling rate real time.

[Colorfingers]

@Mad-Thanos The high definition model is not publicly available and we do not have plans to release. We plan to provide a new update of the Face mesh model in July/Aug if you have a simple filter solution. Please contribute back to the community. Thanks

@mgyong Is this a sales option? Why is the high definition model not going to be released? It has more data points I assume. When will the FaceMesh model updates happen? The models have not been changed for 2 years...

https://github.com/google/mediapipe/tree/master/mediapipe/models

Thanks...

[showbit01]

with 3d landmarks,if we kept photo of a person in front of camera I found the z coordinate of all landmarks are different as the live person in front of camera but it should be same ???

[lakpa-tamang9]

Hi, at this time, did you get any insights about how to remove the jitter to make the animation look smooth? I am also ongoing with the same issue currently.

[UX3D-mazzini]

Did anybody manage to implement the Kalman filter?

[mangguoxia]

我尝试了 OneEuroFilter，看起来它仍然不能很好地减少抖动。 根据我的经验，卡尔曼滤波器是一个公认的选择。 但是过滤FaceMesh坐标会花费很多（468指针太多了）。 如果你只是使用 TransformMatrxi（一个 4x4 矩阵），KalmnaFIlter 会有所帮助。它只有 16 个数字来平滑。 这是一个可参考的方法： 1：初始化 KalmanFilter。 2：得到新的Transform矩阵时，将当前的TransformMatrix与之前的进行对比，如果Matrix变化太大，重置KalmanFilter，使用Raw矩阵。 3：如果矩阵变化小。使用过滤器平滑矩阵。

嘿， 我正在从屏幕地标点获取旋转矩阵。卡尔曼滤波器是一个不错的选择吗？

其实我不能说好不好。我找不到另一种减少抖动的方法。卡尔曼滤波器是我尝试过的最佳选择。

很好，实际上我的用例是从地标点获取头部姿势方向。**** 卡尔曼滤波器对我有用吗？

这是我的应用程序：https ://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063使用卡尔曼滤波器来减少抖动。你可以试试看这个方法是否适合你。

I try the OneEuroFilter, looks like it still not working well to reduce the jitter. From my experience KalmanFIlter is a accepted choice. But It will be much cost to filter the FaceMesh coordinate (468 pointer is too much). If you guy just use the TransformMatrxi (a 4x4 matrix), KalmnaFIlter will be help. It only has 16 number to smooth. This is a referenceable method: 1: Initialize KalmanFilter. 2: When get a new Transform matrix, check current TransformMatrix with previous, If the Matrix change too much, reset KalmanFilter and use the Raw matrix. 3:If matrix change small. Use filter to smooth the Matrix.

Hey, I am working on getting rotation matrix from screen landmarks points.is kalman filter a good choice?

Actually I can't say good or not. I can't find another method to reduce the jitter. Kalman filter is the best choice that I have tried.

That's nice,Actually my use case is to get the head pose orientation from landmarks points.**** Kalman filter is useful for me?

This is my app:https://apps.apple.com/cn/app/%E5%8D%A1%E9%80%9A%E8%84%B8/id1516638063 that use the Kalman filter to reduce the jitter. You can try it to make sure whether the method is usedul for you.

Could you please share your implement of Kalman filter ? I want to know how to use this filter to get more stable landmarks.

[amazingyyc]

Sorry it's not a open source project

[kimzee123]

so how do you use the landmark smoothing calculator? is there any example?

[kuaashish]

Hello @Mad-Thanos, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[skyler14]

With the move to new mediapipe solutions is there a plan to release a high definition model?

[TulgaTDev]

just add kalman filter ez solution

[Fujicsso]

just add kalman filter ez solution

Do you have an example implementation?

[matanox]

There seem to be smoothing functions introduced in the recent source code. It claims to use exponential moving average in/for some of the flows.

(https://github.com/google/mediapipe/blob/496a6ed80905d9fe077f5a4869887cf6f7d5cffe/docs/solutions/pose_classification.md?plain=1#L117)

[Fujicsso]

There seem to be smoothing functions introduced in the recent source code. It claims to use exponential moving average in/for some of the flows.

(

https://github.com/google/mediapipe/blob/496a6ed80905d9fe077f5a4869887cf6f7d5cffe/docs/solutions/pose_classification.md?plain=1#L117 )

So, this appears to be only on the "pose classification" solution, would it be possible to just use that function in the normal Pose solution?