search

aisingapore / TagUI

Free RPA tool by AI Singapore
Apache License 2.0
5.65k stars 585 forks source link

Is there a way to mimic "enter" press - use type or keyboard step with [enter] #504

Closed asakre closed 5 years ago

asakre commented 5 years ago

I need to entersome data in a text/combo box , and simply press enter , as there are no buttons to submit here. Is there a way to add an "enter" press action ?

kensoh commented 5 years ago

Try using type step with [enter] as the text, for eg type identifier as my name[enter]

More details at - https://github.com/kelaberetiv/TagUI#steps-description

For complex keyboard automation, need to use SikuliX automation, more details below -

https://github.com/kelaberetiv/TagUI#visual-automation

asakre commented 5 years ago

I tried the (type identifier as my name[enter]), but didn't work. I searched and found this link which has exactly what I need : https://www.rpa-sg.org/TagUI-Sikuli-Commands/type.php

I tried to run vision type(Key.ENTER) , but it hanged the flow.

kensoh commented 5 years ago

This error usually happens when SikuliX is not set up properly. Check 2 things -

  1. If you have download the packaged installation for your OS from this link https://github.com/kelaberetiv/TagUI#set-up

  2. If you have installed Java JDK v8 (64-bit) or later for the SikuliX automation requirements. The link is found in the visual automation link in previous comment - http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

If above 2 things are checked, you can try again using keyboard step to natively control keyboard.

asakre commented 5 years ago

I have installed the package from the link above, and I've now installed JDK8, and restarted the PC, but still hanging at the vision type(Key.Enter) step. Any ideas ?

image

image

kensoh commented 5 years ago

This is strange issue, have not encounter this. Look for the folder tagui\src\tagui.sikuli. There should be some log file there for Windows. When it hangs, check contents of the log file and paste here.

asakre commented 5 years ago

I think it is case sensitive. I changed it to vision type(Key.ENTER) , and it didn't hang. My problem is that it didn't simulate an "enter" press.

What I simply need is to enter an ID as shown below, which will filter the result to only show 1 result. My problem is selecting this one result.

image

image

click .chosen-single span type .chosen-search input as d1059 vision type(Key.Enter)

Any help would be appreciated.

kensoh commented 5 years ago

I see.. Try using the keyboard step directly. Using vision step to control keyboard is the old way and harder because need to memorise the keywords. For this case it is only enter key, so try below -

  1. try below see if it works for you, assuming pressing enter is how you normally select -

    click .chosen-single span
type .chosen-search input as d1059[enter]

  2. see if you can click inspect on the result so that you can select it directly using its ID or XPath

  3. if you can't, try taking a snapshot of the selection and then visually click the selection by providing the snapshot as what to be selected. eg click selected.png. If the selection text always changes, take the part of screen shot which does not change, for example the right side blue part.

asakre commented 5 years ago

The problem is that, when the robot types in the number, the text box strangely doesn't feel that any text has been typed in. If you see the screenshot below, if I typed this number by myself, it will be filtered as in screenshot 1 , while if robot types it, it will not be filtered (screenshot 2). This is making the issue complicated because now I even can't use Enter, arrow down, tab ..etc. Also I can't use any images or coordinates because the required selection is not filtered.

Screenshot 1 : image

Screenshot 2 : image

I tried Type and Enter along with Xpath, but nothing worked. Any help would be appreciated?

kensoh commented 5 years ago

I see.. this looks like an unusual web control that does not work with the automated event triggers sent to Chrome. For such a case, the best option will be to use the visual automation mode directly.

  1. Take an image snapshot of the text box, include the Select Doctor text so that TagUI can recognise that textbox from other textboxes on the screen. Save image as .png or .bmp format.

  2. TagUI will click on the center of the snapshot in step 1. So make sure the snapshot image is taken so that clicking on the center of the image will trigger a mouse click in the textbox.

  3. Use the following to visually click on the image and enter - 

    type textbox.png as d1059[enter]

  4. If step 3 does not work try below -

    click textbox.png
keyboard d1059[enter]
asakre commented 5 years ago

Thanks Kensoh, the image solution has worked.

asakre commented 5 years ago

I tried to run it in the background (i.e. without chrome) , but it didn't select a doctor. Does the image solution work if I will not use chrome ?

kensoh commented 5 years ago

Oh I'm afraid not. The image solution only works if there is a real browser on the screen for the robot to use computer vision to find and then click / type onto the screen.

If your implementation strictly has to be invisible Chrome mode (headless or default PhantomJS), image solution cannot work. Your other option is using dom step to send custom JavaScript code to simulate the clicking and typing. The level of difficulty can be very hard depending on the website design, and hard for me to advise.

asakre commented 5 years ago

I don't have a problem if the browser is shown, but in such case, can I run more than one instance of this flow ? i.e. I want to initiate this flow multiple times in parallel. If not possiblr, I will check the DOM solution.

kensoh commented 5 years ago

I see... TagUI is not designed to run in parallel. You can try installing TagUI to different directories and run in parallel, but you will probably run into errors. Because it has not been design for parallel runs.

asakre commented 5 years ago

I see. So are there any Free/Open source RPAs out there that could run parallel robots at the same time ? This is a required functionality.

kensoh commented 5 years ago

I'm afraid I'm not aware of any that can do that. Even for paid commercial RPA tools, they run one session per laptop and use a central server to run multiple sessions by using multiple laptops.

You can search around for open-source load-testing tools. They are designed for parallel runs, but they may not work as expected because they do not replicate a real web browser such as Chrome.

ghost commented 5 years ago

Siege: https://www.joedog.org/siege-home/

cosmodeveloper2021 commented 4 years ago

Gracias Kensoh, la solución de imágenes ha funcionado.

Hi, how fixed this?

cosmodeveloper2021 commented 4 years ago

Thanks Kensoh, the image solution has worked.

Hi, How Fixed This?

kensoh commented 3 years ago

Sorry @o3smartcities I was away the past few months - https://github.com/tebelorg/RPA-Python/issues/144

Yes the solution is above - https://github.com/kelaberetiv/TagUI/issues/504#issuecomment-512187840

It works by using visual automation to click on the dropdown, before using keyboard step to type into it.