tebelorg / RPA-Python

Python package for doing RPA
Apache License 2.0
4.85k stars 663 forks source link

Docker Click by image -> Click by wrong coordinates - pending some checks #168

Closed granescb closed 3 years ago

granescb commented 4 years ago

Hello. I tested this functional (click by image) on my work pc and it's worked okay. But when i make the same code inside docker container i got some problem. I tried click by browser extension and RPA tool detect and click by wrong coordinate. For example: Browser extension icon coordinate about x:1786, y:53. If i use this coordinates like rpa.click(1786, 53) - tool works fine. But when i tried to use rpa.click('image.png'), i have click not by icon and after rpa.mouse_xy() i got (1645,76). It's not correct icon coordinate. How to fix it?

kensoh commented 3 years ago

Hi RPA for Python users, ๐Ÿ…—๐Ÿ…๐Ÿ…Ÿ๐Ÿ…Ÿ๐Ÿ…จ โ‘กโ“ชโ‘กโ‘ !

It has been many months since I last worked on this personal side project. Just want to shout out that this project is very much alive, and I hope to look into the 30+ issues raised since July (and bug-fixes if required) soon when I get a moment again.

Below are details if you are interested, why I'm away past few months and why I believe my darkest days are over ๐Ÿ€๐Ÿคž๐Ÿป๐Ÿ™๐Ÿป


Since May last year, I've stopped all work to juggle with my mum's critical illness, taking care of my toddler, and doing housework chores like cleaning and cooking. TBH, my wife and I felt that taking care of a baby and giving her the best nutrition (nursing and home-cooked organic food) and care requires at least 1.5 full-time headcount. There is no life to speak of, other than trying to stay afloat with baby care needs, and keep finding alternative ways to treat my mum, where there're some wins.

Eg importing soy-based tube feeds from Netherlands / Germany / Australia --> after switching away from milk-based tube feeds for 2 weeks she stops requiring dialysis. She started requiring 3x a week dialysis for more than 3 months, after admitting to the hospital. Also, applying herbs like mashed garlic or diced onions to her feet somehow consistently improve her breathing and lung function. Have to keep trying alternative healing methods when her recovery keeps hitting plateau in hospital.

She has been discharged from the hospital a month ago and recovering at home, though very slowly. A longterm medication seems to be one of the contributing causes for her condition. After stopping it, there seems to be improvement to her breathing capacity. She is still bed-bound, but over the course of the year I hope she'll regain strength to move about, and hopefully lungs to recover sufficiently to not require oxygen tank support through a tracheal mask (a hole through her throat).


Since November, I've gone back to work at AI Singapore, a government-funded programme to accelerate AI in Singapore. I'm with Product Engineering team, focusing on RPA and TagUI (open-source RPA tool), which RPA for Python is a wrapper for. There are pretty interesting stuffs on TagUI 2021 roadmap. 2020 was the year which my family was going to move to the UK and work, after I receive its tier 1 exceptional tech talent visa. Then everything changed with the virus outbreak, and my mum's illness. The best thing that happened to me in 2020 was being able to go back to work at AI Singapore - I'm grateful.

I've just hired someone to help with housework, cooking and supporting my wife with baby care. I hope this new arrangement will allow me to deliver on my work at AI Singapore, and soon be able to reply countless friends' messages which circumstances leave me with no choice but go missing for months, and then look into the issues raised here!

Eg of the cost of best-in-class nutrition standard researched and set by wife. Baby just felt restless and broke her plate. Even Corelle's break-resistant plate breaks. Why not use plastic plate? Cuz plastic can leech toxins to the food on it. Why not use Silicon plate? Cuz there is no white silicon materials for plate. Why white? Cuz that helps to improve visibility of the food instead of contrasting against some pink colour or some other colours. I would be spending much time clearing up the mess of 70 over pieces of plate fragments, while baby gets nursed to calm down, if not for our newfound helper.

That's why I feel like my darkest days are over and now seeing glimpses of light at the end of the tunnel.


My vision for RPA for Python remains - I would like it to be the go-to Python package for RPA.

A best-in-class tool in Python users toolbox which you combine with other cool packages to build amazing stuffs.

kensoh commented 3 years ago

Hi @granescb I'm assuming that your Docker Linux container already has OpenCV and Tesseract installed.

Some ideas on next steps -

  1. try using x,y coordinates tool on your Docker instance to read the x,y coordinates to see if they are really different
  2. try out some different images on your laptop and Docker to see if there is a consistent offset (it might be due to different coordinates system on Docker Linux vs your work PC, for example excluding task bar or some offset).
  3. compare the log outputs for computer vision for your PC and Docker, log file is ~/.tagui/src/tagui.sikuli/tagui.log
kensoh commented 3 years ago

Adding on, this RPA package is based on a forked version of TagUI open-source RPA tool.

Free feel to join Telegram group chat for TagUI community to post any questions - https://t.me/rpa_chat