Inaccurate clicking using Sikuli, error in computer vision? - cause unknown yet

[lookang]

I seem to be able to reproduce this error on both windows and mac?

in tagui live snap (200,0)-(250,25) to output.png click output.png

[tagui] INPUT - [11] click /Users/lookang/Desktop/tagui/flows/SLSvle/output.png [tagui] ACTION - click /Users/lookang/Desktop/tagui/flows/SLSvle/output.png [error] RobotDesktop: checkMousePosition: should be L[300,376]@S(0) but after move is L[303,363]@S(0) Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. [log] CLICK on L[300,376]@S(0) (527 msec)

[tagui] OUTPUT - [11] SUCCESS

Instead of clicking on the view (output.png) it clicks somewhere in the middle of the screen. Is it possible something happened in both Windows and Mac that cause it to fail now?

[ruthtxh]

Hi @lookang

Can I check if you have any external monitor connected, if you do can you try to disconnect and only work on your laptop screen?

[lookang]

wow! you are right, disconnect the secondary monitor seems to make the click correctly at the desired position. thanks! will test more, thanks!

[kensoh]

Interesting behaviour. By right, secondary monitor doesn't affect, unless the system menu bar is on the 2nd monitor (Mac) or taskbar on 2nd monitor (Windows). Some info around multi-monitor setups -

https://sikulix-2014.readthedocs.io/en/latest/screen.html#multi-monitor-environments

[lookang]

On my WOG, the tagui.log has error log and it is as follows. snap (0,0)-(100,100) works but the clicking is problematic. It is always clicking some coordinates but not the png. see the snap didnt affect the new position of click. the anti-virus cannot be disable temporary for debugging also. Any ideas how to overcome this?

[error] RobotDesktop: checkMousePosition: should be L[640,360]@S(0) but after move is L[417,240]@S(0) Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. You might try to run the SikuliX stuff as admin. [error] RobotDesktop: checkMousePosition: should be L[960,-540]@S(1) but after move is L[1422,-810]@S(1) Possible cause in case you did not touch the mouse while script was running: Mouse actions are blocked generally or by the frontmost application. You might try to run the SikuliX stuff as admin. [tagui] START - listening for inputs

_[tagui] INPUT - [1] snap (0,0)-(600,600) to C:/rpa/tagui/flows/lookang/login/a.png [tagui] ACTION - snap (0,0)-(600,600) to C:/rpa/tagui/flows/lookang/login/a.png [tagui] OUTPUT - [1] SUCCESS

[tagui] INPUT - [2] click C:/rpa/tagui/flows/lookang/login/a.png [tagui] ACTION - click C:/rpa/tagui/flows/lookang/login/a.png [log] CLICK on L[857,307]@S(0) (535 msec) [tagui] OUTPUT - [2] SUCCESS_

[tagui] INPUT - [3] click C:/rpa/tagui/flows/lookang/login/a.png [tagui] ACTION - click C:/rpa/tagui/flows/lookang/login/a.png [log] CLICK on L[672,307]@S(0) (546 msec) [tagui] OUTPUT - [3] SUCCESS

**[tagui] INPUT - [4] snap (0,0)-(100,100) to C:/rpa/tagui/flows/lookang/login/a.png [tagui] ACTION - snap (0,0)-(100,100) to C:/rpa/tagui/flows/lookang/login/a.png [tagui] OUTPUT - [4] SUCCESS

[tagui] INPUT - [5] click C:/rpa/tagui/flows/lookang/login/a.png [tagui] ACTION - click C:/rpa/tagui/flows/lookang/login/a.png [log] CLICK on L[672,307]@S(0) (543 msec)** [tagui] OUTPUT - [5] SUCCESS

[kensoh]

I've not come across something like this, maybe @ruthtxh can have a look or set up a Zoom call to look into this.

[ruthtxh]

Hi @lookang ,just to confirm, this is ran without an external monitor too right? You can also try to set your windows display settings to 100%.

Based on what I've found on SikuliX forums, there could be a couple of reasons:

1. The app you are automating on could have blocked such control
2. You may need to run as admin for SikuliX to function properly https://answers.launchpad.net/sikuli/+question/674292
3. Also, there could be a bug in SikuliX that could have existed in this version as well. https://github.com/RaiMan/SikuliX1/issues/314

[kensoh]

Fyi @ruthtxh closing some TagUI issues that are around for a while without further inputs.

Probably the last time I do this before starting work. Hi @lookang :wave:, hope this issue has been solved!

[lookang]

In general, the SikuliX works on personal computer.

But on whole of government laptop, it seems SikuliX is blocked when it was working in the beginning of whitelisting

I have evolved my TagUI scripts to be semi automated, needing human clicking in addition, where I am on the WOG laptops.

Maybe it is impossible to debug until I try on my colleague wog laptop when I transfer my files over to him.

[kensoh]

Thanks Lawrence for sharing these! The issue should be reopened. Does the clicking always happen with a specific offset away from where it should click?

I'm thinking of something like this, use hover to move the mouse then use mouse_xy() function to get the coordinates. After that, add the offset before doing a click (x, y)

Other ideas could be always running the browser maximised so that automaton can be done with clicking specific x, y coordinates or an offset of specific x, y.

[kensoh]

Closing but please ping back if this is still relevant or an issue.