Issue accessing elements in frame, for eg click step - fixed, further tests done

[kensoh]

user query, probably not commonly used as this has not been reported.

---

works

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
click //button[.="Search"]

fails

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    click //button[.="Search"]
}

[kensoh]

Thanks @ck81 for your findings! What you said is correct, this seems to be an issue with the behaviour for frame. Can you share with me your past experience with using frame in TagUI? Any such observations, or for your previous use cases, the click was not used?

Some time in June 2018, I switch to using Chrome Puppeteer's best practice by finding x,y location of UI element, moving to the location, and simulating a mouse down and up on that location. When I try your code above, clicking on the search button within a frame or on it's own page shows that the (x,y) is around (1089, 83).

When I try below code to add an offset manually, below code works and the search button is clicked. So it looks like the issue is with calculating the correct location of the UI element. Chrome will return the (x,y) location relative to the frame and not the whole webpage.

But clicking on the (x,y) coordinate as an absolute value base on the whole webpage would not work. A possible solution, if it exists, would be somehow calculate the offset of the frame and factor that into the actual interaction.

js chrome.mouse.action("mousePressed", 1089, 183,"left",1); chrome.mouse.action("mouseReleased", 1089, 183,"left",1);

[kensoh]

Adding technical reference

Doing a snap frame1 yields below in tagui_chrome.log, thus it is accessible, the offset of the frame.

[11] {"id":11,"result":{"result":{"type":"object","value":{"top":83,"left":8,"width":1339,"height":804}}}}

Adding above offset to the earlier test command, the following works to click the search button -

js chrome.mouse.action("mousePressed", 1097, 166,"left",1); chrome.mouse.action("mouseReleased", 1097, 166, "left",1);

[kensoh]

Draft solution for changes in tagui_header.js -

• [x] 1. add new global variables to track the frame offset, eg frame_ste_offset_x and y
• [x] 2. inside chrome.withFrame() get x,y offset of provided frame using chrome.getRect()
• [x] 3. in chrome.getRect() and chrome.mouse.getXY(), add offset to the x,y coordinates returned
• [x] 4. inside chrome.withFrame(), initialise the x,y offset value pair after exiting frame context

Note - in 3, there must be accurate checking whether TagUI is in a frame context, before applying offsets. otherwise this change may impact existing scripts that work on webpages without frames.

[kensoh]

Adding on CK, can you add a dummy name attribute to the frame, eg 'frame_name' or something? I want to test with default PhantomJS mode to verify if the old implementation is working. But adding the attribute myself shows frame with the name not found error with frame step.

(CasperJS framework uses only the name attribute to select the frame to be used, for Chrome integration I use id + name attributes to do the selection).

dom document.querySelector('#frame1').setAttribute("name", "frame_name");

[ck81]

can you add a dummy name attribute to the frame, eg 'frame_name' or something?

Hi Ken, have added the name attribute to the test page: https://rpa-sg.org/tmp/frame1.php

<iframe id="frame1" name="frame_name1"  ...

[kensoh]

Thanks CK, using this I still get the frame not found error. I also wasn't able to find an online sample website with a frame name. To expedite, I'll assume PhantomJS as unknown, and recommend users in future to use headless Chrome with headless option if they require hidden browser for automation and have issues with frames for PhantomJS mode.

frame frame_name1

[kensoh]

Implemented draft solution! Will do some basic and secondary tests.

[kensoh]

Great, following works! Both for within frame and without a frame. I added snap step to test whether the snapshot is also captured accurately. It does. There is minor offset which does not take into account of the border width of the frame.

Will forego optimising this part, I think it is micro-optimising with little to gain in accuracy versus increased risk of issues or failure for some edge cases. For your example, the border width is only 2 pixels.

within a frame

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    snap //button[.="Search"]
    click //button[.="Search"]
}

wait 10

without frame

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
snap //button[.="Search"]
click //button[.="Search"]

wait 10

[kensoh]

Am comfortable to commit the implementation, update now available at cutting edge release -

https://github.com/kelaberetiv/TagUI#set-up

For further testing to make sure fix works (mainframe + subframe and other scenarios) and does not affect functionality of existing scripts (already did a scan of various random sample flows)

[onenessboy]

Am comfortable to commit the implementation, update now available at cutting edge release -

https://github.com/kelaberetiv/TagUI#set-up

For further testing to make sure fix works (mainframe + subframe and other scenarios) and does not affect functionality of existing scripts (already did a scan of various random sample flows)

Hi, I was about to ask question on same. Just a question that chrome extension is not able to write script when i shift from one frame to another.. does above solution fixes extension too.. or just for manual coding ?

[kensoh]

Hi @onenessboy above fix does not change the Chrome extension at all, as there is no access to frame-recording events in the Chrome extension.

[ck81]

Hi Ken,

Sorry for the delay of the testing.

Have downloaded your latest cutting edge release.

Tried your code above on my windows 10 machine. The submit button didn't work for both the versions with frame and without frame:

within a frame

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    snap //button[.="Search"]
    click //button[.="Search"]
}

wait 10

without frame

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
snap //button[.="Search"]
click //button[.="Search"]

wait 10

[ck81]

To be sure, I created another web page to test the frame. The input field is ok. But the submit button didn't work for both.

Without Frame

https://rpa-sg.org/tmp/frames/form1.php
wait 1

enter //input[@name="q"] as test123
snap //input[@id="submitBtn"]
click //input[@id="submitBtn"]

wait 10

With Frame

https://rpa-sg.org/tmp/frames/frame31.php
wait 1

frame frame1
{
    enter //input[@name="q"] as test123
    snap //input[@id="submitBtn"]
    click //input[@id="submitBtn"]
}

wait 10

[ck81]

For further testing to make sure fix works (mainframe + subframe and other scenarios)

Hi Ken,

I've created a web page containing 2 levels of iframes (mainframe + subframe) for your testing purpose: https://rpa-sg.org/tmp/frames/frame32.php

Test script:

https://rpa-sg.org/tmp/frames/frame32.php
wait 1

frame frame2
{
    enter //input[@name="q2"] as this is form2
    snap //input[@id="submitBtn2"]
    click //input[@id="submitBtn2"]
}

frame frame2 | frame1
{
    enter //input[@name="q"] as this is form1
    snap //input[@id="submitBtn"]
    click //input[@id="submitBtn"]
}

wait 10

As highlighted in the previous post, for now on my windows 10 machine, the input field works, but not the submit buttons.

[ck81]

Hi Ken,

Just trying to "stretch" TagUI, I created one webpage with 3 levels of iframes: https://rpa-sg.org/tmp/frames/frame33.php

Test script:

https://rpa-sg.org/tmp/frames/frame33.php
wait 1

frame frame3
{
    enter //input[@name="q3"] as this is form3
    snap //input[@id="submitBtn3"]
    click //input[@id="submitBtn3"]
}

frame frame3 | frame2
{
    enter //input[@name="q2"] as this is form2
    snap //input[@id="submitBtn2"]
    click //input[@id="submitBtn2"]
}

frame frame3 | frame2 | frame1
{
    enter //input[@name="q"] as this is form1
    snap //input[@id="submitBtn"]
    click //input[@id="submitBtn"]
}

wait 10

The first 2 levels work (for the input field), but not Level 3.

Is TagUI supposed to work with more than 2 levels?

[kensoh]

Thanks CK! For your first post, it works on my macOS when I run it with or without frame.

For your third post, the 2 tests on main frame and subframe work when run separately. But when Run consecutively in your above script, there is some change in the webpage after doing the click, thus the search box is no longer present.

For your fourth post, what you observed is correct, TagUI was only developed to work with 2 levels of frames. The code to implement frames are complicated, so I stopped at 2 which should work for most websites I've encountered and so far no user feedback on level 3 frames.

[kensoh]

Looks like there may be some difference in behaviour between macOS and Windows. The TagUI components used are the same, so my initial guess is difference in Chrome behaviour. Can you try if the following works for you? (without the snap steps)

within a frame

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    click //button[.="Search"]
}

wait 10

without frame

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
click //button[.="Search"]

wait 10

[kensoh]

The rationale for trying above is I notice Chrome sometimes behave weirdly when I try to take a snapshot of an element in a frame. It will do the snapshot but later on does not restore the frame back. This isn't something I can control from my side, but now gathering data points, so that if this is consistent, I can recommend users to avoid using snap step with frames.

[kensoh]

If you try the above with snap and it still does not work, can you attach the log in tagui\src\tagui_chrome.log to this issue here by dragging and dropping into the textbox? Then we can compare to see if there is any difference in Chrome's response on Windows and macOS.

Attached below is mine for the test with frame and without snap step - tagui_chrome.log

[ck81]

within a frame

https://rpa-sg.org/tmp/frame2.php
wait 1

frame frame1
{
    enter //input[@name="q"] as RPA
    click //button[.="Search"]
}

wait 10

As before, enter into input field is ok. But click not ok.

Attached please find the tagui_chrome.log file. test3 - with frame - tagui_chrome.log

[ck81]

without frame

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
click //button[.="Search"]

wait 10

Same, enter into input field is ok. But click not ok.

Attached please find the corresponding tagui_chrome.log file. test3 - without frame - tagui_chrome.log

p.s. as highlighted in my email to you, this one works ok if I use your released version of TagUI. But it doesn't work when I use the latest cutting edge release.

[kensoh]

Thanks CK! The logs look the same, except that your computer the x-coordinate is different. The subsequent click correctly adds the same offset on both mine and your computer.

my Mac

[13] {"id":13,"result":{"result":{"type":"object","value":{"x":1089,"y":83}}}}

your Windows

[12] {"id":12,"result":{"result":{"type":"object","value":{"x":1315,"y":83}}}}

We might be experiencing some inconsistent behaviour with Chrome on Windows and macOS. My suspicion now is this - in the cutting edge release there is code to auto-add the x, y offset for the frame, before clicking on the button. This seems like the required solution for the issue initially raised here. But maybe adding offset somehow does not work on Windows, compared to macOS.

I don't have access to a Windows PC now, but I'll see if I can get more clues when I next have access. In the meantime, when you are able to spare some time, can you attach the log here, for the working scenario where you use the packaged release of TagUI? I can compare the response from Chrome with the above to see if there is any clue.

Also, I assume that your browser is at 100% zoom. I don't understand why the x coordinate on your Chrome is more than mine by 200 over pixels. Maybe because the browser resolution is wider or some other reason, that might give more clue.

[ck81]

Also, I assume that your browser is at 100% zoom

No, My laptop has a small screen, so I'm using 125%.

But TagUI works fine at 125% in all previous packaged release - for both xpath and visual automation.

[ck81]

In the meantime, when you are able to spare some time, can you attach the log here, for the working scenario where you use the packaged release of TagUI

As requested, the two log files below are using the packaged release

without frame: both the input field and click button work fine tagui_chrome - without frame - packaged release.log

with frame: the input field works, but not the button tagui_chrome - with frame - packaged release.log

[kensoh]

Hi CK, I've tried running your code for without frame (https://github.com/kelaberetiv/TagUI/issues/553#issuecomment-532533520) and with frame (https://github.com/kelaberetiv/TagUI/issues/553#issuecomment-532532822), on a Windows 10 laptop, and using the cutting edge release. I was unable to replicate the issue that you are facing.

Below is a comparison of your logs, it looks like for some reason, though what TagUI sends to Chrome is the same, for the 2 versions on your laptop, Chrome is returning different x coordinate value. The difference is quite big over 500 pixels. Other than this difference (which affect the subsequent click location), there isn't other differences I notice in your logs.

without frame on your laptop (cutting edge release)

[10] {"id":10,"result":{"result":{"type":"object","value":{"x":1311,"y":83}}}}

without frame on your laptop (packaged release)

[10] {"id":10,"result":{"result":{"type":"object","value":{"x":783,"y":83}}}}

[kensoh]

The only reason that I can think of now may be differences in Chrome settings or user settings.

Could you try if removing the tagui_user_profile folder in tagui\src\chrome in the cutting edge release, and replacing it with the folder from your packaged release helps?

I assume you have installed in separate folders, thus each installation would have its own Chrome user profile and settings. If above works, then it is something to do with the user settings for the 2 profiles, although I don't know what setting yet until you compare to see the difference between both Chrome user settings.

[kensoh]

I'll be attaching the logs for the 2 cases which worked on the Windows 10 laptop I borrowed, for reference and comparison.

[kensoh]

without_frame_borrowed_windows.log

with_frame_borrowed_windows.log

[kensoh]

For both, the typing into the search box works, and the clicking to search and returning search results work. For the without frame there is no offset as expected, while with frame there is offset added accordingly to account of the location of the frame relative to the main document body.

[ck81]

Hi Ken,

To replicate the problem, try setting the screen resolution to 1.25 (for windows 10)

I believe this is the course because I noticed that when I run your cutting edge release, the chrome browser launched by Tagui - the fonts changed to very small.

But when I run using the packaged release, the chrome browser uses my window's default setting.

Chrome is returning different x coordinate value. The difference is quite big over 500 pixels

Think this also explains your observations above.

[kensoh]

I think whether Chrome runs at 125% zoom or 100% zoom is a browser setting which users sets after running it, as TagUI does not adjust the zoom level. So there isn't code change from the packaged release to the cutting edge version which does that.

Most probably, for the cutting edge version, it might have launched when your PC was using a different setting or you have adjust zoom before, thus the setting is saved inside the tagui\src\chrome\tagui_user_profile folder.

I'll try to see if I can replicate the different behaviour using the packaged release and cutting edge version on a borrowed Windows 10 laptop. If it can't be replicated, the next thing is to try the above suggestion to reset your browser setting -

Could you try if removing the tagui_user_profile folder in tagui\src\chrome in the cutting edge release, and replacing it with the folder from your packaged release helps?

[kensoh]

Confirmed, can't replicate what's happening on cutting edge release on your laptop with 125% zoom.

I tried to set the Windows laptop to 125% zoom, when running below script, the text becomes bigger. And both the typing of RPA text and clicking of search button works, showing the search results.

Typing tagui without parameters also show version as v5.6 the cutting edge version. When I tried removing the tagui_user_profile folder for that installation to start fresh, it works too.

https://rpa-sg.org/My-Research-Library/my-research-library.php
wait 1

enter //input[@name="q"] as RPA
click //button[.="Search"]

wait 10

My borrowed Windows 10 laptop returns the following when querying for the Search button -

[11] {"id":11,"result":{"result":{"type":"object","value":{"x":945,"y":83}}}}

Full log for reference - tagui_chrome.log

Since the TagUI version and environment is already synced, I don't have other suggestions to try other than resetting your Chrome user settings by deleting the tagui\src\chrome\tagui_user_profile folder to see if that helps reset some Chrome user settings that may be causing this.

[kensoh]

The last variable is Chrome version, I'm using v76 for both Mac and the Windows.

[ck81]

Hi Ken,

resetting your Chrome user settings by deleting the tagui\src\chrome\tagui_user_profile folder to see if that helps reset some Chrome user settings that may be causing this.

Tried your last suggestion => leads me to discover where the "bug" is.

See, chrome allows us to change the zoom level by pressing Ctrl-+ or Ctrl--. I was doing some testing earlier on a long web page, so my zoom level was set to 80%.

After a few rounds of testing, I think it can be confirmed that TagUI only works when the zoom level for Chrome is set at 100%. Note: it doesn't matter if you set the screen resolution to 100%, 125% or 150%, but the Chrome zoom level has to be set at 100%, otherwise the mouse click will not work.

p.s. you might want to make a note somewhere in your documentation - in case someone make the same mistake as I've made, and think that TagUI is not working

True enough, I used back my old Chrome profile, reset the zoom level to 100%, and everything is working perfectly now!

Have tested using the cutting edge release:

1. All the test script above work now - both with and without frame!

2. Works for both 1-level frame and nested frames (with 2-levels)

3. TagUI works on Windows 10 with different screen resolutions. You can set it at 100%, 125% or 150%. However, make sure the Chrome zoom level is set at 100%.

By the way, I remembered you mentioned that the interim solution is to add a snap() before click() as you do not want to change too many of the original code. However, I found that TagUI runs perfectly even with adding the snap(). Can you please confirm this? Or is it still recommended to add the snap() before click()

Yes, think you can close this issue. Frame now works perfectly for TagUI. Thanks a lot for your time and effort, Ken! iframes is very common in a lot of websites. Many RPA tools do not have a good and clean way to handle this. I still find your TagUI the most elegant way in handling frames, by simply specifying the frame name followed by the 2 curly brackets. And it even works with nested iframes - very cool!

[ck81]

Hi Ken,

For your convenience, have set up a web page for you to test a 2-level nested iframes: https://rpa-sg.org/tmp/frames/frame32.php

Sample test script for the nested iframe:

https://rpa-sg.org/tmp/frames/frame32.php
wait 1

frame frame2
{
    enter //input[@name="q2"] as this is form2
    click //input[@id="submitBtn2"]
}
wait 3

frame frame2 | frame1
{
    enter //input[@name="q"] as this is form1
    click //input[@id="submitBtn"]
}
wait 10

[ck81]

Have also set up another simpler page to test a 1-level iframe: https://rpa-sg.org/tmp/frames/frame31.php

https://rpa-sg.org/tmp/frames/frame31.php
wait 1

frame frame1
{
    enter //input[@name="q"] as test123
    //snap //input[@id="submitBtn"]
    //wait 1
    click //input[@id="submitBtn"]
}
wait 10

[kensoh]

Thank you very much CK for the patience and exhaustive testing, and also setting up test pages for testing!! Glad finally found the root cause of the different behaviour on your laptop and mine.

Actually I didn't know about this, and so far no users has feedback the use case of using Chrome at a zoom level different from 100%. Yes this is important to be documented, I raised issue #573 to update readme at an appropriate location to warn about this. Will definitely save a lot of frustration for users not using the zoom level at 100%.

For below paragraph, can you tell me the context of when I said that? For above examples in this issue, I used snap for verification purpose only, that the updated code is now able to handle frame offset correctly and doing snapshot at the correct region of the webpage. So it is definitely ok to take out the snap step before doing the click step.

By the way, I remembered you mentioned that the interim solution is to add a snap() before click() as you do not want to change too many of the original code. However, I found that TagUI runs perfectly even with adding the snap(). Can you please confirm this? Or is it still recommended to add the snap() before click()

Lastly, frames can be very tricky, if you run into issues dealing with frames, let me know again and we'll look into it together to figure out a solution or fine-tune the existing implementation!