Currently facial pose affects the cursor position. This will be a problem for facial gestures (#25) if every time you try to click it moves the cursor.
First of all, this could be solved by using the face pose directly, if it were fast and accurate enough. This would also allow rotation to move the cursor more naturally, as right now, using points tracked on the surface of your face as projected in the 2D camera image, movement slows and stops as the point reaches the edge of your projected face, or in other words as the tangent of your face's surface where the point is tracked becomes parallel to the camera. In other other words, the movement of the theoretical 3D point that it's tracking becomes parallel to the camera as you rotate your head 90 degrees, and this parallel movement (depth) is not tracked.
Secondly, this could be solved by tracking a point on the head unaffected by facial gestures. Faces are, however, incredibly expressive! I see only a few candidates:
Above the forehead, but I think hair is a problem for the tracking. It's too 3D/complex/noisy.
The center of the bridge of the nose. Sadly, this causes problems when the head rotates side to side.
The top of the bridge of the nose, as a compromise. This definitely can be affected by eyebrow raises and nose scrunching. I already try to add a tracking point here. I also add points elsewhere, because it's useful to have redundancy (since points can lose tracking and disappear at any time), but it should be better to use the nose bridge point exclusively when it exists, and only use other points if it doesn't.
Oh, I just thought of a third option. Like the first, use the head rotation as tracked by facemesh, but since it's slow, use it only for movement not otherwise handled by the tracking points. Calculate the movement to be done by rotation and subtract the movement to theoretically already done by translation by looking at either the tracking points's movement since the last facemesh update, or some projected points (based on the last facemesh result and the new one) — not sure which. This is probably a terrible idea though. It technically provides the best of both worlds, but in a likely confusing, unpleasant mixture.
Currently facial pose affects the cursor position. This will be a problem for facial gestures (#25) if every time you try to click it moves the cursor.
First of all, this could be solved by using the face pose directly, if it were fast and accurate enough. This would also allow rotation to move the cursor more naturally, as right now, using points tracked on the surface of your face as projected in the 2D camera image, movement slows and stops as the point reaches the edge of your projected face, or in other words as the tangent of your face's surface where the point is tracked becomes parallel to the camera. In other other words, the movement of the theoretical 3D point that it's tracking becomes parallel to the camera as you rotate your head 90 degrees, and this parallel movement (depth) is not tracked.
Secondly, this could be solved by tracking a point on the head unaffected by facial gestures. Faces are, however, incredibly expressive! I see only a few candidates:
Oh, I just thought of a third option. Like the first, use the head rotation as tracked by facemesh, but since it's slow, use it only for movement not otherwise handled by the tracking points. Calculate the movement to be done by rotation and subtract the movement to theoretically already done by translation by looking at either the tracking points's movement since the last facemesh update, or some projected points (based on the last facemesh result and the new one) — not sure which. This is probably a terrible idea though. It technically provides the best of both worlds, but in a likely confusing, unpleasant mixture.