search

aisingapore / TagUI

Free RPA tool by AI Singapore
Apache License 2.0
5.64k stars 585 forks source link

Visual automation using screenshot not working - issue ard Oracle Java #628

Closed xiaoyinglim closed 4 years ago

xiaoyinglim commented 4 years ago

I used the click "image name".png command to click on items on the website for visual automation. The click command worked a few days also but now it is not working. My friend is also having the same problem. We already tried to redo the ss and made sure that the resolution was 100% or the recommended one. May I know how to solve this issue? Attached are my file codes. [Uploading SGX.zip…]()

kensoh commented 4 years ago

Hi @xiaoyinglim the zip attachment has not been uploaded yet. You can try clicking above there is nothing to download. Try uploading again as a comment, wait for the upload status to complete before submitting the comment.

kensoh commented 4 years ago

In normal situations, if nothing changes, an automation script will not suddenly not work. Without further details, it might be something changed but what has changed is not yet known. For eg the website may be rendering the UI elements differently, or perhaps the company IT policy may start blocking some outgoing web traffic, or maybe prevent write access to the folder used by TagUI (similar to an issue faced by your friend the other time when using the tool).

Lion666 commented 4 years ago

Hi, I got a similar issue. Actually I wanted to update you with a specific post when I fell on this post. I finally understood what the problem was. When I had a concomitant citrix session active that seem to mess up with the coordinates found by the algorithm to locate the screenshot. If I restart my computer and make sure there is no concomitant citrix session the program works fine again. Just stopping the citrix session is not enough I had to restart the machine.

kensoh commented 4 years ago

Hi Lionel, thanks for sharing the solution that works for you! :)

Hi @xiaoyinglim other than uploading the attachment, you can also try restarting computer to see if that works. In addition, in your attachment, include the log files found under the src\tagui.sikuli folder. It contains debugging info for the computer vision engine. Also, let me know your laptop screen resolution. I assume the display zoom is at 100%.

xiaoyinglim commented 4 years ago

Hi Mr Ken, I am unable to upload the files from my work computer. Is it ok if I get back to you tomorrow?

kensoh commented 4 years ago

Oic.. Ok got it! If your company network blocks uploading to GitHub, you can try if using Firefox Send (https://send.firefox.com) works and post the link here.

kensoh commented 4 years ago

Adding on, if you can find out why your computer network blocks the upload to GitHub it will be great. Because it might be the same internet access issue that is the root cause with clicking.

xiaoyinglim commented 4 years ago

These are my files and the Desktop Resolution is 1366 x 768. SGX.zip

xiaoyinglim commented 4 years ago

https://send.firefox.com/download/47a0d74ea039f8d2/#q9n4AlHXmtcZrz3TYgxRxA

kensoh commented 4 years ago

Thanks! Some observations -

  1. inside tagui.log, the response by SikuliX visual automation engine seems alright, and matches the screen resolution on your computer -

    [tagui] INPUT  - [1] click c:/tagui/src/samples/maximise.png
[tagui] ACTION - click c:/tagui/src/samples/maximise.png
[log] CLICK on L[1286,20]@S(0) (1095 msec)
[tagui] OUTPUT - [1] SUCCESS

  2. the run.log you attach is invalid. it is for live mode, and not the actual run where the error happens. can you paste the contents of your run.log showing the error encountered?

START - automation started - Wed Nov 13 2019 17:24:16 GMT+0800 (Malay Peninsula Standard Time)
https://www2.sgx.com/securities/annual-reports-related-documents - Singapore Exchange Ltd

LIVE MODE - type done to quit
  1. another thing you have to tell me exactly which click you are referring to below which does not work. otherwise i can't compare with the logs to help with troubleshooting the issue. and you probably have to share the tagui.log again because the one you sent only have the first click on maximising, which I assume it works.
click maximise.png
click (//*[@class="sgx-icon sgx-select-filter-icon"])[2]
click annual.png
click (//*[@id="from"])
click (//*[@class="sgx-datepicker-toggle-view"])
click previous.png
click december.png
click 31.png
click company name.png
kensoh commented 4 years ago

Adding on, you mentioned that it works on your non-work laptop but not on the work laptop. In addition to the updated run.log and tagui.log and exactly which click fails, can you also attach the same logs for your non-work laptop? So that both you and I can compare to look for more clues.

xiaoyinglim commented 4 years ago

Hi Mr Ken, I have checked my codes by running the file already and all the click png command cannot work. Althought the click maximise.png was able to run in the command prompt, the click did not register on the webpage. The update files are in https://send.firefox.com/download/99fde5da2f458582/#iGSwYeBWxLg3BCDgyOFKAg same password.

xiaoyinglim commented 4 years ago

I will send you my files that I run on my personal laptop when I get home.

kensoh commented 4 years ago

In your above attachment, the tagui.sikuli logs are all empty, and the run.log only has the following lines. If the maximise can run but not click, try to put 2 lines of the maximise, because the first click need to ensure that the chrome browser is in focus and foreground. just like how you manually need to click on the browser before it responds to further clicks.

START - automation started - Tue Nov 19 2019 10:22:10 GMT+0800 (Malay Peninsula Standard Time)
https://www2.sgx.com/securities/annual-reports-related-documents - Singapore Exchange Ltd

wait 10 seconds
click c:/tagui/src/samples/maximise.png
xiaoyinglim commented 4 years ago

My bad, the click maximise.png cannot run in command prompt either. When the code run, it shows these two lines and stops there. I tried putting two lines of maximise too but it did not work.

START - automation started - Tue Nov 19 2019 10:22:10 GMT+0800 (Malay Peninsula Standard Time) https://www2.sgx.com/securities/annual-reports-related-documents - Singapore Exchange Ltd

wait 10 seconds click c:/tagui/src/samples/maximise.png

kensoh commented 4 years ago

Ok got it, when it hangs, can you paste the contents of the log files in src\tagui.sikuli here? Let me see if there is any error messages coming from the SikuliX visual automation engine.

kensoh commented 4 years ago

After you send the info above to me here, try the following -

  1. install Amazon Corretto v8 64-bit from this link - https://aws.amazon.com/corretto
  2. restart your computer to see if the visual automation works

Oracle Java has stopped supporting Java for commercial usage. So it could be somehow Java blocking usage on your computer knowing that you are using for commercial purpose. Above will install the open-source and free version of Java that can be used for business.

xiaoyinglim commented 4 years ago

Ok got it, when it hangs, can you paste the contents of the log files in src\tagui.sikuli here? Let me see if there is any error messages coming from the SikuliX visual automation engine.

There are no contents in the log file though.. :( It's empty.

kensoh commented 4 years ago

If contents log are empty for SikuliX means something wrong either with Java or the installation.

Do the following to find out what's wrong -

  1. install Amazon Corretto as above 2 steps
  2. repeat the setup process of TagUI to unzip then overwrite with the cutting edge release - https://github.com/kelaberetiv/TagUI#set-up

I assume you haven't simply copy the whole TagUI folder to another computer and hope that it works, because it doesn't. There is a Jython file included in the SikuliX automation package which will be removed after first run. If you copy the whole folder to another computer, this file will be missing and SikuliX won't work. Need to do the step 2 above.

xiaoyinglim commented 4 years ago

After you send the info above to me here, try the following -

  1. install Amazon Corretto v8 64-bit from this link - https://aws.amazon.com/corretto
  2. restart your computer to see if the visual automation works

Oracle Java has stopped supporting Java for commercial usage. So it could be somehow Java blocking usage on your computer knowing that you are using for commercial purpose. Above will install the open-source and free version of Java that can be used for business.

OMG installing Amazon Cornetto solves the problem!! Thank you Mr Ken! I did not thought of downloading it because previously when I run my codes the click png command work.

kensoh commented 4 years ago

Oh great!! I guess now we figure out the root cause. Somehow the Java on your company work laptop suddenly stop working. It could be Oracle trying to block usage because you are using for business purpose, or it could be your company IT policy changes and blocks usage of Oracle Java.

xiaoyinglim commented 4 years ago

If contents log are empty for SikuliX means something wrong either with Java or the installation.

Do the following to find out what's wrong -

  1. install Amazon Corretto as above 2 steps
  2. repeat the setup process of TagUI to unzip then overwrite with the cutting edge release - https://github.com/kelaberetiv/TagUI#set-up

I assume you haven't simply copy the whole TagUI folder to another computer and hope that it works, because it doesn't. There is a Jython file included in the SikuliX automation package which will be removed after first run. If you copy the whole folder to another computer, this file will be missing and SikuliX won't work. Need to do the step 2 above.

No I did not copy the folder over to my other computer. I installed the whole package from the set-up page to the computer haha

kensoh commented 4 years ago

Great finally found the root cause!! - https://github.com/kelaberetiv/TagUI/issues/628#issuecomment-555363593