search

kennyhml / pyinterception

Modern Python port & wrapper for the interception device driver
MIT License
124 stars 26 forks source link

Mouse movement with Bezier curve does not work properly. #37

Closed Emre-Kahveci closed 3 days ago

Emre-Kahveci commented 4 days ago

During movement with the Bezier curve, the mouse approaches the desired point from a certain point but cannot reach the target. Also, after the library is run, there is a change in the mouse speed.

Test environment (No intervention has been made to the libraries and they have been newly installed): 

curve_params = interception.beziercurve.BezierCurveParams()
interception.beziercurve.set_default_params(curve_params)

def move_and_print_position(x, y):
    interception.move_to(x, y)
    sleep(.3)
    print(f"Position after going to {x}, {y}:  {interception.mouse_position()}")

print("First Position: ", interception.mouse_position())
move_and_print_position(500, 500)
move_and_print_position(800, 800)
move_and_print_position(500, 500)

output:

First Position:  (296, 422)
Position after going to 500, 500:  (423, 466)
Position after going to 800, 800:  (657, 674)
Position after going to 500, 500:  (561, 570)
kennyhml commented 4 days ago

Try to add a sleep at the end of the with block here to give the OS more time to process the inputs before changing mouse accel again. Let me know how that goes.

Emre-Kahveci commented 4 days ago

I added 0.5 seconds of waiting (time.sleep(0.5) the result remained almost unchanged. sleep code:

    ...
    with _utils.disable_mouse_acceleration():
        time.sleep(0.5)
        for point in curve.points:
        ...

output:

First Position:  (295, 281)
Position after going to 500, 500:  (417, 415)
Position after going to 800, 800:  (654, 657)
Position after going to 500, 500:  (558, 560)
Emre-Kahveci commented 4 days ago

I solved the problem and sent a merge request. The solution worked for me. Can you check? Output after solution:

First Position:  (299, 338)
Position after going to 500, 500:  (499, 499)
Position after going to 800, 800:  (799, 799)
Position after going to 500, 500:  (499, 499)
kennyhml commented 4 days ago

The PR isn't a solution. Verify you have the sleep placed correctly. Also try removing the with block entirely and disabling enhanced pointer precision yourself.

Emre-Kahveci commented 4 days ago

I shared the Sleep code above. I am sure that it is placed correctly, because a cooldown of 0.5 seconds can be felt before the movement. As in the PR I shared, I updated the library and my problem was solved. Now it works smoothly

kennyhml commented 4 days ago

As stated in the PR, your changes dont solve anything. The entire point of the curve movement is to appear human. Manual mouse input doesnt use absolute movements, it uses relative movements. Performing a curve movement with absolute movements is no different than just performing an absolute movement to the destination point in terms of detectability.

Here are my results adding a 0.5s sleep:

Position after going to 500, 500:  (498, 497)
Position after going to 800, 800:  (798, 800)
Position after going to 500, 500:  (500, 500)

Definitely not completely accurate yes. But significantly better than your results were. Adding a 0.1s sleep between each movement:

Position after going to 500, 500:  (500, 500)
Position after going to 800, 800:  (800, 800)
Position after going to 500, 500:  (500, 500)

It is pixel-perfect, which suggests the slight inaccuracy is caused by the speed it is going. I will try to find a more optimal middle ground of speed and accuracy, but this should be something you should be able to play around with and find out for yourself perfectly fine.

kennyhml commented 4 days ago

In summary, this isnt an issue with the logic of the movements or the type of movements itself. Inaccuracy appears to be happening due to speed and a more optimal balance needs to be found. My guess is that the 75% chance to sleep might be what is fucking it up causing every 4th movement on average to be executed too fast and somehow resulting in inaccuracies. One thing that is sort of unfortunate is that python doesnt really have the ability to sleep in the low milliseconds not to mention microseconds.

Emre-Kahveci commented 2 days ago

To find the source of the problem, I updated the code to print the values. (there are 100 lines of output because I print at each step)

edited move_to code:

curve = beziercurve.HumanCurve(mouse_position(), _utils.normalize(x, y))

# Track where we currently are since we need to round the points generated
# which will, especially on longer curves, offset us if we dont adjust.

curr_x, curr_y = curve.points[0]
last_real_x, last_real_y = mouse_position()

# Mouse acceleration must be disabled to preserve precision on relative movements
with _utils.disable_mouse_acceleration():
    time.sleep(1)
    for point in curve.points:
        rel_x: int = round(point[0] - curr_x)
        rel_y: int = round(point[1] - curr_y)

        real_curr_x, real_curr_y = mouse_position()
        print(f"cp:({curr_x}, {curr_y})\treal cp: ({real_curr_x}, {real_curr_y})\trel p:({rel_x}, {rel_y})\t", end="")

        curr_x, curr_y = curr_x + rel_x, curr_y + rel_y

        stroke = MouseStroke(MouseFlag.MOUSE_MOVE_RELATIVE, 0, 0, rel_x, rel_y)
        _g_context.send(_g_context.mouse, stroke)

        time.sleep(0.2)
        real_x, real_y = mouse_position()

        print(f"real p:{real_x, real_y}\tmove:({real_x - last_real_x}, {real_y - last_real_y})\t", end="")
        print(f"rel-move diff x:{rel_x-(real_x - last_real_x)} y:{rel_y-(real_y - last_real_y)}")
        last_real_x, last_real_y = real_x, real_y

        time.sleep(0.05)

cp = current position (according to the code, the mouse should be here) real cp = mouse's coordinate (before rel moves) rel p = rel values real p = coordinate of the mouse after the movement move = change made by the mouse in the last move rel-move diff = difference between movement and rel values ​​(actually should be 0)

output:

cp:(100.0, 101.0)   real cp: (100, 101) rel p:(0, 0)    real p:(100, 101)   move:(0, 0) rel-move diff x:0 y:0
cp:(100.0, 101.0)   real cp: (100, 101) rel p:(17, 9)   real p:(110, 106)   move:(10, 5)    rel-move diff x:7 y:4
cp:(117.0, 110.0)   real cp: (110, 106) rel p:(14, 6)   real p:(119, 110)   move:(9, 4) rel-move diff x:5 y:2
cp:(131.0, 116.0)   real cp: (119, 110) rel p:(16, 7)   real p:(129, 114)   move:(10, 4)    rel-move diff x:6 y:3
cp:(147.0, 123.0)   real cp: (129, 114) rel p:(16, 10)  real p:(139, 121)   move:(10, 7)    rel-move diff x:6 y:3
cp:(163.0, 133.0)   real cp: (139, 121) rel p:(16, 8)   real p:(149, 126)   move:(10, 5)    rel-move diff x:6 y:3
cp:(179.0, 141.0)   real cp: (149, 126) rel p:(13, 8)   real p:(157, 131)   move:(8, 5) rel-move diff x:5 y:3
cp:(192.0, 149.0)   real cp: (157, 131) rel p:(15, 11)  real p:(166, 137)   move:(9, 6) rel-move diff x:6 y:5
cp:(207.0, 160.0)   real cp: (166, 137) rel p:(12, 8)   real p:(174, 142)   move:(8, 5) rel-move diff x:4 y:3
cp:(219.0, 168.0)   real cp: (174, 142) rel p:(14, 11)  real p:(183, 149)   move:(9, 7) rel-move diff x:5 y:4
cp:(233.0, 179.0)   real cp: (183, 149) rel p:(13, 12)  real p:(191, 157)   move:(8, 8) rel-move diff x:5 y:4
cp:(246.0, 191.0)   real cp: (191, 157) rel p:(11, 8)   real p:(198, 162)   move:(7, 5) rel-move diff x:4 y:3
cp:(257.0, 199.0)   real cp: (198, 162) rel p:(12, 11)  real p:(205, 169)   move:(7, 7) rel-move diff x:5 y:4
cp:(269.0, 210.0)   real cp: (205, 169) rel p:(11, 10)  real p:(212, 175)   move:(7, 6) rel-move diff x:4 y:4
cp:(280.0, 220.0)   real cp: (212, 175) rel p:(11, 10)  real p:(219, 181)   move:(7, 6) rel-move diff x:4 y:4
cp:(291.0, 230.0)   real cp: (219, 181) rel p:(11, 11)  real p:(226, 188)   move:(7, 7) rel-move diff x:4 y:4
cp:(302.0, 241.0)   real cp: (226, 188) rel p:(10, 11)  real p:(232, 195)   move:(6, 7) rel-move diff x:4 y:4
cp:(312.0, 252.0)   real cp: (232, 195) rel p:(11, 11)  real p:(239, 202)   move:(7, 7) rel-move diff x:4 y:4
cp:(323.0, 263.0)   real cp: (239, 202) rel p:(8, 9)    real p:(244, 207)   move:(5, 5) rel-move diff x:3 y:4
cp:(331.0, 272.0)   real cp: (244, 207) rel p:(10, 13)  real p:(250, 216)   move:(6, 9) rel-move diff x:4 y:4
cp:(341.0, 285.0)   real cp: (250, 216) rel p:(8, 9)    real p:(255, 221)   move:(5, 5) rel-move diff x:3 y:4
cp:(349.0, 294.0)   real cp: (255, 221) rel p:(9, 11)   real p:(261, 228)   move:(6, 7) rel-move diff x:3 y:4
cp:(358.0, 305.0)   real cp: (261, 228) rel p:(8, 9)    real p:(266, 234)   move:(5, 6) rel-move diff x:3 y:3
cp:(366.0, 314.0)   real cp: (266, 234) rel p:(7, 9)    real p:(270, 239)   move:(4, 5) rel-move diff x:3 y:4
cp:(373.0, 323.0)   real cp: (270, 239) rel p:(9, 11)   real p:(276, 246)   move:(6, 7) rel-move diff x:3 y:4
cp:(382.0, 334.0)   real cp: (276, 246) rel p:(7, 9)    real p:(280, 252)   move:(4, 6) rel-move diff x:3 y:3
cp:(389.0, 343.0)   real cp: (280, 252) rel p:(6, 9)    real p:(284, 257)   move:(4, 5) rel-move diff x:2 y:4
cp:(395.0, 352.0)   real cp: (284, 257) rel p:(6, 8)    real p:(288, 262)   move:(4, 5) rel-move diff x:2 y:3
cp:(401.0, 360.0)   real cp: (288, 262) rel p:(6, 9)    real p:(291, 268)   move:(3, 6) rel-move diff x:3 y:3
cp:(407.0, 369.0)   real cp: (291, 268) rel p:(6, 9)    real p:(295, 274)   move:(4, 6) rel-move diff x:2 y:3
cp:(413.0, 378.0)   real cp: (295, 274) rel p:(6, 9)    real p:(299, 279)   move:(4, 5) rel-move diff x:2 y:4
cp:(419.0, 387.0)   real cp: (299, 279) rel p:(5, 7)    real p:(302, 284)   move:(3, 5) rel-move diff x:2 y:2
cp:(424.0, 394.0)   real cp: (302, 284) rel p:(6, 8)    real p:(306, 289)   move:(4, 5) rel-move diff x:2 y:3
cp:(430.0, 402.0)   real cp: (306, 289) rel p:(4, 9)    real p:(308, 294)   move:(2, 5) rel-move diff x:2 y:4
cp:(434.0, 411.0)   real cp: (308, 294) rel p:(5, 5)    real p:(311, 297)   move:(3, 3) rel-move diff x:2 y:2
cp:(439.0, 416.0)   real cp: (311, 297) rel p:(5, 8)    real p:(315, 302)   move:(4, 5) rel-move diff x:1 y:3
cp:(444.0, 424.0)   real cp: (315, 302) rel p:(4, 7)    real p:(317, 307)   move:(2, 5) rel-move diff x:2 y:2
cp:(448.0, 431.0)   real cp: (317, 307) rel p:(4, 6)    real p:(320, 311)   move:(3, 4) rel-move diff x:1 y:2
cp:(452.0, 437.0)   real cp: (320, 311) rel p:(4, 6)    real p:(322, 314)   move:(2, 3) rel-move diff x:2 y:3
cp:(456.0, 443.0)   real cp: (322, 314) rel p:(3, 9)    real p:(324, 320)   move:(2, 6) rel-move diff x:1 y:3
cp:(459.0, 452.0)   real cp: (324, 320) rel p:(4, 3)    real p:(326, 322)   move:(2, 2) rel-move diff x:2 y:1
cp:(463.0, 455.0)   real cp: (326, 322) rel p:(3, 8)    real p:(328, 327)   move:(2, 5) rel-move diff x:1 y:3
cp:(466.0, 463.0)   real cp: (328, 327) rel p:(3, 2)    real p:(330, 328)   move:(2, 1) rel-move diff x:1 y:1
cp:(469.0, 465.0)   real cp: (330, 328) rel p:(3, 4)    real p:(332, 331)   move:(2, 3) rel-move diff x:1 y:1
cp:(472.0, 469.0)   real cp: (332, 331) rel p:(2, 7)    real p:(333, 335)   move:(1, 4) rel-move diff x:1 y:3
cp:(474.0, 476.0)   real cp: (333, 335) rel p:(3, 2)    real p:(335, 336)   move:(2, 1) rel-move diff x:1 y:1
cp:(477.0, 478.0)   real cp: (335, 336) rel p:(2, 4)    real p:(336, 339)   move:(1, 3) rel-move diff x:1 y:1
cp:(479.0, 482.0)   real cp: (336, 339) rel p:(2, 3)    real p:(338, 341)   move:(2, 2) rel-move diff x:0 y:1
cp:(481.0, 485.0)   real cp: (338, 341) rel p:(3, 4)    real p:(340, 343)   move:(2, 2) rel-move diff x:1 y:2
cp:(484.0, 489.0)   real cp: (340, 343) rel p:(1, 4)    real p:(340, 346)   move:(0, 3) rel-move diff x:1 y:1
cp:(485.0, 493.0)   real cp: (340, 346) rel p:(2, 3)    real p:(341, 347)   move:(1, 1) rel-move diff x:1 y:2
cp:(487.0, 496.0)   real cp: (341, 347) rel p:(2, 3)    real p:(343, 349)   move:(2, 2) rel-move diff x:0 y:1
cp:(489.0, 499.0)   real cp: (343, 349) rel p:(1, 5)    real p:(343, 352)   move:(0, 3) rel-move diff x:1 y:2
cp:(490.0, 504.0)   real cp: (343, 352) rel p:(1, -1)   real p:(344, 352)   move:(1, 0) rel-move diff x:0 y:-1
cp:(491.0, 503.0)   real cp: (344, 352) rel p:(2, 3)    real p:(345, 354)   move:(1, 2) rel-move diff x:1 y:1
cp:(493.0, 506.0)   real cp: (345, 354) rel p:(1, 2)    real p:(346, 355)   move:(1, 1) rel-move diff x:0 y:1
cp:(494.0, 508.0)   real cp: (346, 355) rel p:(1, 0)    real p:(346, 355)   move:(0, 0) rel-move diff x:1 y:0
cp:(495.0, 508.0)   real cp: (346, 355) rel p:(1, 3)    real p:(347, 357)   move:(1, 2) rel-move diff x:0 y:1
cp:(496.0, 511.0)   real cp: (347, 357) rel p:(1, 2)    real p:(348, 358)   move:(1, 1) rel-move diff x:0 y:1
cp:(497.0, 513.0)   real cp: (348, 358) rel p:(1, 1)    real p:(348, 359)   move:(0, 1) rel-move diff x:1 y:0
cp:(498.0, 514.0)   real cp: (348, 359) rel p:(1, -1)   real p:(349, 359)   move:(1, 0) rel-move diff x:0 y:-1
cp:(499.0, 513.0)   real cp: (349, 359) rel p:(0, 1)    real p:(349, 359)   move:(0, 0) rel-move diff x:0 y:1
cp:(499.0, 514.0)   real cp: (349, 359) rel p:(1, 0)    real p:(350, 359)   move:(1, 0) rel-move diff x:0 y:0
cp:(500.0, 514.0)   real cp: (350, 359) rel p:(0, 1)    real p:(350, 359)   move:(0, 0) rel-move diff x:0 y:1
cp:(500.0, 515.0)   real cp: (350, 359) rel p:(1, 1)    real p:(350, 360)   move:(0, 1) rel-move diff x:1 y:0
cp:(501.0, 516.0)   real cp: (350, 360) rel p:(0, 1)    real p:(350, 361)   move:(0, 1) rel-move diff x:0 y:0
cp:(501.0, 517.0)   real cp: (350, 361) rel p:(0, -1)   real p:(350, 361)   move:(0, 0) rel-move diff x:0 y:-1
cp:(501.0, 516.0)   real cp: (350, 361) rel p:(1, -1)   real p:(351, 360)   move:(1, -1)    rel-move diff x:0 y:0
cp:(502.0, 515.0)   real cp: (351, 360) rel p:(0, 0)    real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 515.0)   real cp: (351, 360) rel p:(0, 0)    real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 515.0)   real cp: (351, 360) rel p:(0, 0)    real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 515.0)   real cp: (351, 360) rel p:(0, -1)   real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:-1
cp:(502.0, 514.0)   real cp: (351, 360) rel p:(0, 0)    real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 514.0)   real cp: (351, 360) rel p:(0, 0)    real p:(351, 360)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 514.0)   real cp: (351, 360) rel p:(0, -1)   real p:(351, 359)   move:(0, -1)    rel-move diff x:0 y:0
cp:(502.0, 513.0)   real cp: (351, 359) rel p:(0, -1)   real p:(351, 358)   move:(0, -1)    rel-move diff x:0 y:0
cp:(502.0, 512.0)   real cp: (351, 358) rel p:(0, 0)    real p:(351, 358)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 512.0)   real cp: (351, 358) rel p:(0, -1)   real p:(351, 358)   move:(0, 0) rel-move diff x:0 y:-1
cp:(502.0, 511.0)   real cp: (351, 358) rel p:(0, -1)   real p:(351, 357)   move:(0, -1)    rel-move diff x:0 y:0
cp:(502.0, 510.0)   real cp: (351, 357) rel p:(0, 0)    real p:(351, 357)   move:(0, 0) rel-move diff x:0 y:0
cp:(502.0, 510.0)   real cp: (351, 357) rel p:(0, 1)    real p:(351, 357)   move:(0, 0) rel-move diff x:0 y:1
cp:(502.0, 511.0)   real cp: (351, 357) rel p:(0, -2)   real p:(351, 356)   move:(0, -1)    rel-move diff x:0 y:-1
cp:(502.0, 509.0)   real cp: (351, 356) rel p:(0, -2)   real p:(351, 355)   move:(0, -1)    rel-move diff x:0 y:-1
cp:(502.0, 507.0)   real cp: (351, 355) rel p:(-1, 0)   real p:(351, 355)   move:(0, 0) rel-move diff x:-1 y:0
cp:(501.0, 507.0)   real cp: (351, 355) rel p:(0, 1)    real p:(351, 355)   move:(0, 0) rel-move diff x:0 y:1
cp:(501.0, 508.0)   real cp: (351, 355) rel p:(0, -3)   real p:(351, 354)   move:(0, -1)    rel-move diff x:0 y:-2
cp:(501.0, 505.0)   real cp: (351, 354) rel p:(0, 0)    real p:(351, 354)   move:(0, 0) rel-move diff x:0 y:0
cp:(501.0, 505.0)   real cp: (351, 354) rel p:(0, 1)    real p:(351, 354)   move:(0, 0) rel-move diff x:0 y:1
cp:(501.0, 506.0)   real cp: (351, 354) rel p:(0, -3)   real p:(351, 353)   move:(0, -1)    rel-move diff x:0 y:-2
cp:(501.0, 503.0)   real cp: (351, 353) rel p:(0, 0)    real p:(351, 353)   move:(0, 0) rel-move diff x:0 y:0
cp:(501.0, 503.0)   real cp: (351, 353) rel p:(0, 2)    real p:(351, 353)   move:(0, 0) rel-move diff x:0 y:2
cp:(501.0, 505.0)   real cp: (351, 353) rel p:(-1, -2)  real p:(350, 353)   move:(-1, 0)    rel-move diff x:0 y:-2
cp:(500.0, 503.0)   real cp: (350, 353) rel p:(0, 1)    real p:(350, 353)   move:(0, 0) rel-move diff x:0 y:1
cp:(500.0, 504.0)   real cp: (350, 353) rel p:(0, 0)    real p:(350, 353)   move:(0, 0) rel-move diff x:0 y:0
cp:(500.0, 504.0)   real cp: (350, 353) rel p:(0, 0)    real p:(350, 353)   move:(0, 0) rel-move diff x:0 y:0
cp:(500.0, 504.0)   real cp: (350, 353) rel p:(0, -3)   real p:(350, 351)   move:(0, -2)    rel-move diff x:0 y:-1
cp:(500.0, 501.0)   real cp: (350, 351) rel p:(0, 0)    real p:(350, 351)   move:(0, 0) rel-move diff x:0 y:0
cp:(500.0, 501.0)   real cp: (350, 351) rel p:(0, 0)    real p:(350, 351)   move:(0, 0) rel-move diff x:0 y:0
cp:(500.0, 501.0)   real cp: (350, 351) rel p:(0, 0)    real p:(350, 351)   move:(0, 0) rel-move diff x:0 y:0
cp:(500.0, 501.0)   real cp: (350, 351) rel p:(0, -1)   real p:(350, 351)   move:(0, 0) rel-move diff x:0 y:-1
(350, 351)

The mouse does not move as far as the code moves. I tried waiting 0.5 seconds after each mouse movement (mouse movement took about 1 minute) but still the result did not change.

kennyhml commented 8 hours ago

Correct me if im wrong, but that just means it was off by a single pixel no?