zkghost commented 5 years ago

Context

I am writing a program to control mouse movement using your project (via autohotpy) and have run into some issues getting the mouse to go where I expect it.

The code to move a mouse in Autohotpy is as follows:

    def moveMouseToPosition(self, x, y):
        width_constant = 65535.0/float(self.user32.GetSystemMetrics(0)) # metric: main monitor width
        height_constant = 65535.0/float(self.user32.GetSystemMetrics (1)) # metric: main monitor height
        # move mouse to the specified position
        stroke = InterceptionMouseStroke()
        stroke.state = InterceptionMouseState.INTERCEPTION_MOUSE_MOVE
        stroke.flags = InterceptionMouseFlag.INTERCEPTION_MOUSE_MOVE_ABSOLUTE
        stroke.x = int(float(x)*width_constant)
        stroke.y = int(float(y)*height_constant)
        self.sendToDefaultMouse(stroke)

    def sendToDefaultMouse(self, stroke):
        self.interception.interception_send(self.context, self.default_mouse_device, ctypes.byref(stroke), 1)

    def interception_send(self, context, device, stroke_p, nstroke):
        """
        #int ITERCEPTION_API interception_send(InterceptionContext context, InterceptionDevice device, const InterceptionStroke *stroke, unsigned int nstroke);
        """
        return self.interceptionDll.interception_send(context, device, stroke_p, nstroke)

My monitor setup is something like this:

                              main
+----------------------+ +-------------+ +-------------+
|    3840 x 2160       | |             | |             |
|                      | |  1920x1200  | |  1920x1200  |
|                      | |             | |             |
|                      | +-------------+ +-------------+
|                      |
|                      |
+----------------------+

So a 4k monitor on the left, two 1920x1200 monitors on the right, and the middle monitor is the primary display.

I discovered that I can't just enter the x and y values for a stroke directly to move the mouse to that pixel (ie: if call moveMouseToPosition(500,500) and then query the position of the mouse, it is actually somewhere else.

This works on a single monitor setup, however. I am glad that works but would like my program to work regardless of user monitor setup.

I pieced together that the constant factor for mouse movement 65535.0 represents all pixels across a dimension (width/height). So when I want to move to say, x: 960 y: 0 (middle top of the primary monitor), what I have to do is:

Find where the middle monitor starts and ends along the x-axis: 3840 (offset from the first 4k monitor) and 5760 (3840 + the 1920 width of the primary monitor
Find the half way point between those bounds (5760-3840)/2 + 3840 = 4800
Find the proportion of total pixels along all screens that the 4800 point represents 4800/(3840+1920+1920) = 0.625
Use the proportion above to find the proportion of 65535 I want to target: 65535 * 0.625 = 40959.375
Solve x * (65535/1920) = 40959.375 (x times scaling factor over main monitor width = target proportion of 65535): 40959.375/(65535.0/1920) = 1200

If I input moveMouseToPosition(1200, 0), then it correctly places it at pixel 960 on the main monitor (in the middle of the 1920 pixel width).

Problem

I've found I can't do the same in the Y axis. If I hop through the same gymnastics (though it is much easier because my monitors aren't stacked vertically), I should be able to do:

600 (middle of monitor along y-axis) = 32767.5 (half of 65535) / (65535/1200 (main monitor height))

But inputting moveMouseToPosition(0, 600) doesn't place me in the middle of the main monitor, it places me at the bottom (x: 0, y: 1079).

The code I'm using to determine mouse location after I've moved it via the inception driver:

    def getMousePosition(self):
        #x, y = win32api.GetCursorPos()
        res = Point()
        self.user32.GetCursorPos(ctypes.byref(res))
        return (res.x,res.y)

Here's the output of targeting different positions along the Y-axis in 120 pixel increments (1/10th of 1200), where I'm trying to do the same math above on the input to transform it into the proper value to be the right value in the output space:

--------
moving to x: 0, y: 0
mouse moved to: (0, 0)
x: 0/1920 = 0.0
y: 0/1200 = 0.0 (Y: 0) (Input Y / Output Y: div_by_0
--------
moving to x: 0, y: 120
mouse moved to: (0, 215)
x: 0/1920 = 0.0
y: 215/1200 = 0.17916666666666667 (Y: 120) (Input Y / Output Y): 0.5581395348837209
--------
moving to x: 0, y: 240
mouse moved to: (0, 431)
x: 0/1920 = 0.0
y: 431/1200 = 0.3591666666666667 (Y: 240) (Input Y / Output Y): 0.5568445475638051
--------
moving to x: 0, y: 360
mouse moved to: (0, 647)
x: 0/1920 = 0.0
y: 647/1200 = 0.5391666666666667 (Y: 360) (Input Y / Output Y): 0.5564142194744977
--------
moving to x: 0, y: 480
mouse moved to: (0, 863)
x: 0/1920 = 0.0
y: 863/1200 = 0.7191666666666666 (Y: 480) (Input Y / Output Y): 0.5561993047508691
--------
moving to x: 0, y: 600
mouse moved to: (0, 1079)
x: 0/1920 = 0.0
y: 1079/1200 = 0.8991666666666667 (Y: 600) (Input Y / Output Y): 0.5560704355885079

You can see that the actual output mouse location is far off from the input pixel, and the ratio between the input value and output location differs for different inputs.

Just looking at the output, it looks like I could get the y value close to the intended value by multiplying the output by 11/20 (55%). But then my project becomes very specialized for my scenario, which I'd rather avoid.

I guess I could also do some calibrating by doing mouse movements and measuring the actual location the mouse ends up at, and fitting the coefficient that way, but that seems like a lot of unnecessary overhead.

Questions

Is the behavior I'm seeing intended? Known?
Is my understanding of how to send the mouse to a particular location correct?
How do I get the mouse to go to a specific x/y coordinate on the screen? At least the main monitor?

oblitum commented 5 years ago

Mouse absolute positioning is like you did (old post mentioning this detail), see for example:

https://github.com/oblitum/Interception/blob/77e71d96f544577e439c1ec353530d90c670e019/samples/mathpointer/mathpointer.cpp#L146-L147

But I really never tried a monitor setup like yours, so at the moment, I can tell from experience the expected behavior.

zkghost commented 5 years ago

Hmm, I see. I was actually playing around, I think I found out at least how to get the input coords to translate successfully into the absolute positioning space

transformation_math(desired_y_pixel) * (main_monitor_height / largest_monitor_height - 1)

So in my case, 1200 (from my main 1920x1200 monitor) / 2159 (from my 4K monitor).

It is sort of annoying though, because when I query the system for all the monitors and their bounding boxes, the pixels given are post DPI scaling, ie:

# left top right bottom
[-3840, 0, -2304, 864

The left/right coordinates are negative because they are left of my main monitor, but you'll notice it goes from 3840 to 2304, instead of going from 3840 to 0; that is due to the 250% scaling on that monitor.

Detecting the scaling is a little tricky (for example, I can't detect the scaling on the Y-axis in my scenario because my monitors are arranged horizontally), but I think I can do it.

Anyways, I'll go ahead and close this and leave this here for anyone else who hits a similar issue. Thanks for the response!

oblitum / Interception

Mouse Movement along Y axis not working? #84

Context

Problem

Questions