robotjs donot move right distance when screen resolution changes dynamicly

Javonmeng commented 3 years ago

After the robotjs process is running, change the screen resolution and let robotjs move the mouse the same pixel distance. For example, if robotjs moves from (0, 0) to (1366, 100), the mouse will move to the same screen position. It seems that robots will calculate the moving distance based on the initial resolution of the process, rather than the real-time screen resolution

Expected Behavior

The mouse cursor should move to a different position on screen according to the set resolution

Current Behavior

The mouse cursor moves to the same position on screen

Possible Solution

when run movemouse(), robotjs should reacquire current screen resolution

Steps to Reproduce (for bugs)

set screen resolution to 1920x1080
start a process (keep running using "for loop")
robot.moveMouse(1366, 100);
reset screen resolution to 1366x768
robot.moveMouse(1366, 100);
you will see mouse position is not at the screen side

Context

Your Environment

RobotJS version: 0.6.0 Node.js version:14.16.0 npm version: 7.19.1 Operating System: Windows 10 Electron: 13.1.4

jason-crawford-xio commented 3 years ago

I too can reproduce this.

jason-crawford-xio commented 3 years ago

It looks like removing this line of the code should work around the problem by simply having the method always use the current value of the virtual screen size rather than the cached value.

jason-crawford-xio commented 3 years ago

@oktapodia Are there any complications to consider regarding that one line change? (Obviously at one point someone felt there was good reason to cache the screen config.)

ragauskl commented 3 years ago

The issue is in this line, SendInput should be called on separate thread that has call to SetThreadDesktop (separate thread because it won't always work on current thread),

As SendInput sends mouse event, remarks from SetCursorPos apply, which says "The input desktop must be the current desktop when you call SetCursorPos"

I don't quite understand how it all works in detail, but solution below worked for me in custom native module (c++):

Note - Code below might have bad conventions etc, I'm just starting with c/c++ and use it for limited personal needs
// mouse.cc
typedef struct _MouseMoveEvent {
int x;
int y;
HDESK hwnd;
} MouseMoveEvent;

DWORD WINAPI moveMouseOnThread(LPVOID lpParam) { string logMsg; MouseMoveEvent e = (MouseMoveEvent)lpParam;

bool success; success = SwitchDesktop(e->hwnd);

if (!success) { DWORD e = GetLastError(); // error handle }

success = SetThreadDesktop(e->hwnd);

if (!success) { DWORD e = GetLastError(); // error handle }

size_t x = MOUSE_COORD_TO_ABS(e->x - vScreenMinX, vScreenWidth); size_t y = MOUSE_COORD_TO_ABS(e->y - vScreenMinY, vScreenHeight);

INPUT mouseInput = {0}; mouseInput.type = INPUT_MOUSE; mouseInput.mi.dx = x; mouseInput.mi.dy = y; mouseInput.mi.dwFlags = MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_MOVE | MOUSEEVENTF_VIRTUALDESK; mouseInput.mi.time = 0;

SendInput(1, &mouseInput, sizeof(mouseInput));

return 0; }

void moveMouse(int x, int y) { if(vScreenWidth<0 || vScreenHeight<0) updateScreenMetrics();

HDESK hOldDesktop = GetThreadDesktop(GetCurrentThreadId()); HDESK hwnd = OpenInputDesktop(0, true, GENERIC_ALL);

MouseMoveEvent data = (MouseMoveEvent) malloc( sizeof(MouseMoveEvent) );

data->x = x; data->y = y; data->hwnd = hwnd;

DWORD dwThreadId; HANDLE hThread = CreateThread( NULL, 0, moveMouseOnThread, data, 0, &dwThreadId);

if (hThread == NULL) { DWORD e = GetLastError(); // error handle return; }

DWORD r = WaitForSingleObject(hThread, INFINITE); if (r != 0) { // error handle }

CloseHandle(hThread);

// not sure if this line is required on this thread SwitchDesktop(hOldDesktop); }



If I'll find some time I might submit a pull request to add this fix to the repo, if no one will have it done by that time

octalmage / robotjs