what's the unit of the cooridinates in this line code?

[lucasjinreal]

it was very small, is it pixels differs on image pixel?

[chenjiunhan]

The range of x and y are from 0.0 to 1.0 within the whole image. For example, if the resolution of an image is 1920x1080, so the difference of x between adjacent pixels in x axis will be 1.0/1920 ~= 0.00052

[lucasjinreal]

@chenjiunhan I think it was normalize towards input pose model image wh right? But I set cv2.cap width and height, and I found the keypoints on image to the rightest border, the x coordinates only 0.3, it should about 0.9 since as you said it was normalized to 0-1.

Is it start from left top corner?

[chenjiunhan]

It starts from left top corner. However, demo.py Line 73 cv2.imshow('HAC demo', cv2.flip(image, 1)) plots a horizontally flipped image, which is easier for us to understand our movement. Because it looks just like a mirror reflecting our bodies. Many applications, like discord, zoom...etc, they flip camera images for us, but actually other people see the image without flipped.

The pose model receives an image without flipped as the input, your right hand is close to the leftmost, so the value of x coordinate is close to 0.3. cv2.imshow('HAC demo', cv2.flip(image, 1)) flipped the image for you, that's why you thought it should be around 0.9, however, the value was small than you expected.

It's confusing indeed, thank you for pointing that out. 😄

[lucasjinreal]

@chenjiunhan thank u for your detailed explaination. I get it now. I am trying using index finger as movement driver of pointer, since I found the center point of palm is really easy to tired if you hang your hand in the air for a long time.

So that it was much more easier if we can controll pointer with finger movement rather than your whole hand movement.

However, no luck....

Did you considered this aspect?

Oh, btw, did you also thought about removing bouncing of tiny movements when you try to focus on a certain point? I am still try eliminate this drawback, what do u think?

[lucasjinreal]

@chenjiunhan Also, did u thinked about add multi-task quick gesture in model? Such as switch desktoop on windows, such as open task-manager inferface ? this is useful and much more less accuracy required.

[chenjiunhan]

@chenjiunhan thank u for your detailed explaination. I get it now. I am trying using index finger as movement driver of pointer, since I found the center point of palm is really easy to tired if you hang your hand in the air for a long time.

So that it was much more easier if we can controll pointer with finger movement rather than your whole hand movement.

However, no luck....

Did you considered this aspect?

Oh, btw, did you also thought about removing bouncing of tiny movements when you try to focus on a certain point? I am still try eliminate this drawback, what do u think?

I analyzed jittering tiny movements a bit. It turns out using the wrist as a fixed point to move mouse is a bad idea. The wrist has very high jittering distance among key points of the hand. And index finger has the lowest jittering distance between frames. r_i_d_x means x coordinate of Right hand Index finger Dip

The labels of y axis are defined by Mediapipe.

As you mentioned, using index finger might be a better choice to reduce the jittering. I used wrist it's because when I grab a fist, the wrist doesn't move too much. So, it really depends how we design the mapping between gestures and mouse control. If we want to use index finger to control the movement of mouse, we need to figure out a way to click and drag, at the same time, it keeps the position of cursor.

@jinfagang

[chenjiunhan]

@chenjiunhan thank u for your detailed explaination. I get it now. I am trying using index finger as movement driver of pointer, since I found the center point of palm is really easy to tired if you hang your hand in the air for a long time.

So that it was much more easier if we can controll pointer with finger movement rather than your whole hand movement.

However, no luck....

Did you considered this aspect?

Oh, btw, did you also thought about removing bouncing of tiny movements when you try to focus on a certain point? I am still try eliminate this drawback, what do u think?

If you want to move mouse with the index finger, you have to

1. collect data with recoding.py
2. define the action set in conf/action_sets
3. retrain the model with train.py

check out this for more details: recording-custom-actions

[chenjiunhan]

Trajectories of each frame of key points, I tried not to move XD...

[lucasjinreal]

@chenjiunhan Yeah, you are right. Using index finger is not only from the jitter distance minimal consideration but also for user experienrances since it is much more better to move your finger than move your wrist.

As for mouse control, for left click, I think it need 3d dimension distance measure but this doesn;t supported in mediapipe.

But if we want make it work on 2D, we can measure static status, which is your index finger static for about 500ms, then you will set control to status waiting for click, if users keep moveing then wipe out this status and keep cycle.

in this way, users can static on a point, and then grad fist to click or drag. As for finger movement along with fist, it can filter out by a time interval.

Do u have a better idea?

[chenjiunhan]

Hi @jinfagang,

I had a covid-19 vaccine recently..., But I'm back.

I tried different designs inspired by you and Mihail, an improved version fast & slow mode can be seen in the following gif:

Using the index finger is a fast mode, to move mouse cursor faster. Opening the palm is a slow mode, which can control mouse accurately. The others are the same.

It seems that it's hard to use other key points except for the wrist to control mouse stably, during changing different gestures, wrist is the only one without moving.

[chenjiunhan]

I've pushed the code to main, and the demo code can be seen here

[lucasjinreal]

@chenjiunhan thank u, i'll try it out! seems more accurate than before.

what do u think is the limitations now in current version? I mean in your daily usage, what's the limitations.