frame() and popup() - explore technical feasibility and hacks [done]

[ck81]

Hi Ken,

Have been using frame() and popup() in TagUI on some websites that use iFrame and popup windows. They worked amazingly well! And these 2 commands are another unique TagUI features that are not available in many other RPA tools.

Was wondering if these two commands have been ported to TagUI-Python?

[kensoh]

Hey CK, these 2 commands are not ported, because they are not supported in TagUI live mode. The syntax is such that user defines the code block and then insert the steps in between to be done. But in live mode, each statement is executed 1 by 1, making this impossible.

But I think having this capabilities will open up more use cases to TagUI for Python. Will take a look to see if there is some hack (eg using custom JavaScript + chrome_step with send()) that is technically possible to implement.

[ck81]

Hi Ken,

Just some thought...

(1) Not sure if you can modify TagUI a bit such that it can also run the tagui command in LIVE mode. This means that we can store some TagUI scripts in a file and then call this script from LIVE mode.

(2) If [1] is possible, then you can add one more TagUI-Python command e.g. tagui(). This means we can now extend TagUI-Python to run almost any TagUI scripts and commands from python, including frame() and popup().

[kensoh]

For 1, this is not technically feasible, because tagui step actually expands out the script. But within that script there can be steps such as frame / popup / run / check etc which not supported in live mode. This will create unexpected behaviour and errors.

For 2, actually if you have an existing installation of TagUI, you can use the run() function to do so already now, frame and popup will work, but it will be a separate TagUI process.

[kensoh]

I'm looking if it is possible to find a hack to do this. The catch with implementing such a hack at TagUI main project would be inconsistent behaviour with PhantomJS and Firefox mode because I'm hacking the Chrome communications.

If the hack exists, this catch is ok in TagUI for Python as only Chrome is the browser used.

[kensoh]

An update that frame() POC works and coming right up!

Reference - tested with https://www.w3schools.com/html/html_iframe.asp for main frame and https://internet-banking.dbs.com.sg/IB/Welcome for main frame + sub frame

---

popup() POC works and it is also on the way.

Reference - tested with http://tebel.org to click on links and use title() and snap() on popup tabs

[kensoh]

Implemented and available with pip install tagui --upgrade

Documentation at https://github.com/tebelorg/TagUI-Python#pro-functions

Sample usage for frame() with a frame on webpage

# set context to the webpage frame
t.frame('mainframe_name_or_id')

# do your stuffs on elements in that frame

# exit frame context and reset to main webpage
t.frame()

Sample usage for frame() on a frame within a frame

# set context to the webpage frame
t.frame('mainframe_name_or_id', 'subframe_name_or_id')

# do your stuffs on elements in that subframe

# exit frame context and reset to main webpage
t.frame()

Sample usage for popup() on a new popup tab launched by main page

# set context to the webpage popup tab
t.popup('unique_string_in_url_of_popup_tab')

# do your stuffs on elements on that webpage popup tab

# exit popup context and reset to main webpage
t.popup()

[ck81]

Implemented and available with pip install tagui --upgrade

You're amazing, Ken!

Let me try this on some websites tomorrow and get back to you on the test result.

[kensoh]

Thanks CK, hopefully the implementation is ok🤞- look forward to hearing from your testing!

[kensoh]

Hi CK, a note that I did a minor update to v1.11.1. There was an upstream patch for TagUI for live mode for variables to work correctly. This patch should not cause impact to TagUI for Python, and I have already tested.

Fyi in case you have not started testing frame() and popup(), then upgrade to this version (v1.11.1 instead of v1.11.0) before testing is preferred, because this will be the codebase going forward.

[ck81]

Hi Ken,

Have downloaded the latest version v1.11.1

Tested on 2 of my corporate websites with iframe().

1. type() works perfect!

2. But for some reason, click() didn't work. I have tested by loading the url within the iframe. It tested ok. So there's nothing wrong with the xpath or the click command. Have tested clicking on a number of elements. All didn't work. This is the same for both websites.

[ck81]

Have created a website for your easy testing: https://rpa-sg.org/tmp/frame1.php

The webpage contains only 2 lines:

<h3>iFrame Test Page</h3>

<iframe id="frame1" style="width: 100%; height: 800px; overflow: hidden;" src="https://rpa-sg.org/My-Research-Library/my-research-library.php" width="100" height="100" scrolling="no">Iframes not supported</iframe>

It loads the following page into iframe (with the id "frame1"): https://rpa-sg.org/My-Research-Library/my-research-library.php

It has many HTML elements to test type() and click()

As highlighted above, the following works:

>>> t.frame('frame1')
True
>>> t.type('//input[@name="q"]', 'RPA')
True

But the following didn't work, even though it returns True:

>>> t.click('//button[.="Search"]')
True

[ck81]

Just to be sure, tried running the same thing using TagUI and found something interesting.

First, try running the TagUI script below:

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
click //button[.="Search"]

You will see that the script runs perfectly - it enters the search term "RPA" then clicks the Search button.

Now try to run the following script:

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    click //button[.="Search"]
}

The above actually gives the same results as TagUI-Python script, that is, you will see "RPA" typed into the search field, but the Search button is not clicked!

My guess is that the bug may be in TagUI and not at the python integration, or your hack to make frame() work for TagUI-Python.

[kensoh]

Thanks CK for your findings! What you said is correct, this seems to be an issue with the behaviour in TagUI main project. Can you share with me your past experience with using frame in TagUI? Any such observations, or for your previous use cases, the click was not used?

Some time in June 2018, I switch to using Chrome Puppeteer's best practice by finding x,y location of UI element, moving to the location, and simulating a mouse down and up on that location. When I try your code above, clicking on the search button within a frame or on it's own page shows that the (x,y) is around (1089, 83).

When I try below code to add an offset manually, below code works and the search button is clicked. So it looks like the issue is with calculating the correct location of the UI element. Chrome will return the (x,y) location relative to the frame and not the whole webpage. But clicking on the (x,y) coordinate as an absolute value base on the whole webpage would not work.

t.send('js chrome.mouse.action("mousePressed", 1089, 183,"left",1); chrome.mouse.action("mouseReleased", 1089, 183,"left",1)')

What I'll do is raise an issue over at the TagUI main project for further investigation. A possible solution, if it exists, would be somehow calculate the offset of the frame and factor that into the actual interaction. In the meantime, I'll also look out for any findings on popup()!

[kensoh]

Issue https://github.com/kelaberetiv/TagUI/issues/553 created as it is an issue with the original TagUI project, and copied you.

[ck81]

Can you share with me your past experience with using frame in TagUI? Any such observations, or for your previous use cases, the click was not used?

Hi Ken,

A lot of our corporate web pages uses iframe. Although using the frame() command works, as I've highlighted in many of my previous posts, I use LIVE mode a lot to do testing and debugging. As frame() does not run in LIVE mode, I always used the method that you've shared with me before: 1) Grab the url of the web page of the iframe 2) Go to the page in (1) directly

As such, I've actually not used frame() extensively in the past. That's why I never bumped into this problem before.

However, I encountered one corporate website recently that uses iframe within iframe! I managed to grab the first level of iframe. But for the 2nd level iframe, it actually use iframe to display a popup calendar for selecting date. I've seen many websites that uses popup window for date selection. It's the first time I see websites using iframe for this. Since it's a date selection that will pipe the selected date back to the parent window, it doesn't make sense to get the url of the 2nd level iframe and go to that directly. So I'm "forced" to use the frame() command. That's when I discovered that click() doesn't seem to work properly. The date cannot be selected using click().

Not sure if this is additional clues to you. I actually tried using dom() to click, it doesn't work either. Note: Yes, the dom() is also issued within frame()

[kensoh]

Thanks CK yes all these details are useful to work towards the solution!

In the interim, maybe using keyboard or visual automation can work for this calendar popup scenario. TagUI can recognise an image anchor (some unique part of the calendar popup) and is able to let user provide x and y coordinates offset (for eg clicking next month or previous month button).

[ck81]

In the interim, maybe using keyboard or visual automation can work for this calendar popup scenario

Yes, I know visual automation can always work as last resort. But I always come back to your TagUI again and again because I find that your TagUI is still the best tool when it comes to web-based automation. TagUI's support for standard xpath, coupled with the ability to use dom(), allows me to handle almost any kind of web-based applications!

[kensoh]

Synced upstream code-fix logic for frame step, closing issue for now. Since issue is with upstream, it'll be tracked from the upstream issue.

Reference sample test Python scripts

within a frame

import tagui as t
t.init()
t.url('https://rpa-sg.org/tmp/frame2.php')
# wait to ensure frame contents are loaded before switching
t.wait()
t.frame('frame1')
t.type('//input[@name="q"]', 'RPA')
t.snap('//button[.="Search"]','button.png')
t.click('//button[.="Search"]')
t.frame()
# wait to see search results before closing
t.wait()
t.close()

without a frame

import tagui as t
t.init()
t.url('https://rpa-sg.org/My-Research-Library/my-research-library.php')
# no wait is needed, will search for element until timeout
t.type('//input[@name="q"]', 'RPA')
t.snap('//button[.="Search"]','button.png')
t.click('//button[.="Search"]')
# wait to see search results before closing
t.wait()
t.close()

[ck81]

Hi Ken,

Now that frame() works for nested iframes (https://github.com/kelaberetiv/TagUI/issues/553),

Have just tested the same thing on TagUI-Python. It works perfectly too!

Thanks for adding the frame() support for TagUI-Python!

p.s. found that click() works too without snap(). However, your sample code above always precedes the click() with snap(). Is this absolutely necessary?

[ck81]

Sample script for testing single iframe:

import tagui as t
t.init()

t.url('https://rpa-sg.org/tmp/frames/frame31.php')
t.wait(1)

t.frame('frame1')
t.type('//input[@name="q"]', 'test123')
t.click('//input[@id="submitBtn"]')
t.frame()

[ck81]

Sample script for testing nested iframes (2 levels):

import tagui as t
t.init()

t.url('https://rpa-sg.org/tmp/frames/frame32.php')
t.wait(1)

t.frame('frame2')
t.type('//input[@name="q2"]', 'this is form2')
t.click('//input[@id="submitBtn2"]')
t.frame()
t.wait(3)

t.frame('frame2', 'frame1')
t.type('//input[@name="q"]', 'this is form1')
t.click('//input[@id="submitBtn"]')
t.frame()

p.s. for those users migrating over from TagUI, sub-frame is written as

t.frame('frame2', 'frame1')

and not

t.frame('frame2 | frame1')

[kensoh]

Thank you CK for raising this issue and now other users benefit with the frame() and popup() functions. Also, thank you very much for the test websites you created in order to test the features.

For below, I use snap() for verification only - to confirm that x,y coordinates offset are handled correctly. If the frame offset is handled correctly, snap will will capture the screenshot fo the web element correctly. It's definitely ok to click() without first using snap().

p.s. found that click() works too without snap(). However, your sample code above always precedes the click() with snap(). Is this absolutely necessary?