Click action for "_WD_ElementAction" function does not work if the element is not visible in the window

demdante commented 1 year ago

Bug report

Describe the bug

As the subject of the issue states, if you try to click on an element on a webpage using "_WD_ElementAction", the function will not work if the element is not visible on the web page. The main example that I want to focus on for this particular issue is when the browser needs to scroll down for the element to be visible.

How to reproduce

Consider this autoit script


#include <wd_helper.au3>
#include <wd_capabilities.au3>
#include <Array.au3>
#include <File.au3>
#include <Inet.au3>
_WD_Option ( "Driver", "C:\tools\geckodriver.exe" )
_WD_Option('DriverParams', '--log trace')
_WD_Option('Port', 4444)
_WD_CapabilitiesStartup()
_WD_CapabilitiesAdd('alwaysMatch', 'firefox')
_WD_CapabilitiesAdd('browserName', 'firefox')
_WD_CapabilitiesAdd('acceptInsecureCerts', True)
_WD_CapabilitiesAdd('javascriptEnabled', True)
_WD_CapabilitiesAdd('acceptInsecureCerts', True)
$sCapabilities = _WD_CapabilitiesGet()
_WD_Startup()
If @error Then Exit SetError(2, @error)
Global $sSession = _WD_CreateSession($sCapabilities)
_WD_Navigate ( $sSession, "https://storeurll.z13.web.core.windows.net/wh.html" )
_WD_LoadWait ( $sSession )
Sleep ( 2000 )
$greet = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_Subject"]' )
$text = _WD_ElementAction ( $sSession, $greet, "VALUE", "Contact the CEO" )
$sal = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_Prefix"]' )
_WD_ElementAction ( $sSession, $sal, "VALUE", "Mr." )
$first = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_FirstName"]' )
_WD_SetElementValue ( $sSession, $first, "firstname" )
$middle = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_middle_name"]' )
_WD_SetElementValue ( $sSession, $middle, "middlename" )
$last = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_lastName"]' )
_WD_SetElementValue ( $sSession, $last, "lastname" )
$email = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_email_address"]' )
_WD_SetElementValue ( $sSession, $email, "mymemail@gmail.com" )
$phonenum = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_phone"]' )
_WD_SetElementValue ( $sSession, $phonenum, "5555555555" )
$strt = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_Street"]' )
_WD_SetElementValue ( $sSession, $strt, "809 Fake Address" )
$cit = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_City"]' )
_WD_SetElementValue ( $sSession, $cit, "Austin" )
$stat = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_State"]' )
_WD_ElementAction ( $sSession, $stat, "VALUE", "Texas" )
$zi = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_zipcode"]' )
_WD_SetElementValue ( $sSession, $zi, "30369" )
$body = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_message"]' )
_WD_SetElementValue ( $sSession, $body, "Test message" )
$sub = _WD_FindElement ( $sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_22"]' )
; This first attempt to click the button will fail, as the command attempts to click the button BEFORE scrolling down the page so that it is visible in the window
$theclick = _WD_ElementAction ( $sSession, $sub, "CLICK" )
MsgBox ( 1, "", $theclick )
; This second attempt to click the button will work since the previous command scrolled the element into view after attempting the click
$theclick = _WD_ElementAction ( $sSession, $sub, "CLICK" )
MsgBox ( 1, "", $theclick )
_WD_DeleteSession ( $sSession )
_WD_Shutdown ()

; It seems to me that there needs to be logic included in the "_WD_ElementAction" command to ensure that the element is visible in the window before attempting to ; perform the action on it, like automatically scrolling the browser window so that the element is visible BEFORE the function attempts to execute the action ; against it. Another product that I use quite frequently that is really good at automating web browsers, UI.Vision RPA Software (https://ui.vision/rpa) is able ; to run the exact same automation that is attempted above, but it is able to successfully click the button the first time that it tries because it implements the behavior ; I described earlier. I also bring up this particular product because it, too, is open source (https://github.com/A9T9/RPA), so I figured that the source code of a ; product that actually works might help you in fixing yours (I think the src\common\command_runner.js file is the one you need).


3. Execute the above autoit script on your machine
4. As you will see, this autoit script is trying to automate filling out an online form, and it works fine all the way until the end, when the script attempts to submit the form by clicking the submit button.  The script will display a message box each time that it attempts to click the submit button.  As you will see, the first attempt will fail, but the second will succeed.

### Expected behavior

A clear and concise description of what you expected to happen.

### Screenshots

[See here](https://reccloud.com/u/ga85oww)

### Additional context

It seems to me that there needs to be logic included in the "_WD_ElementAction" command to ensure that the element is visible in the window before attempting to perform the action on it, like automatically scrolling the browser window so that the element is visible BEFORE the function attempts to execute the action against it.  Another product that I use quite frequently that is really good at automating web browsers, UI.Vision RPA Software (https://ui.vision/rpa) is able to run the exact same automation that is attempted above, but it is able to successfully click the button the first time that it tries because it implements the behavior I described earlier.  I also bring up this particular product because it, too, is open source (https://github.com/A9T9/RPA), so I figured that the source code of a product that actually works might help you in fixing yours (I think the src\common\command_runner.js file is the one you need).  

### System under test

Please complete the following information.

- OS: Windows 11
- OS Arch.: X64
- Browser: firefox
- Browser version: 107.0.1 (64-bit)

Danp2 commented 1 year ago

Thanks for the detailed report. From my understanding of the Webdriver specs, the element should automatically be scrolled into view before the subsequent action is attempted. FWIW, this appears to be a timing issue as the 2nd click will also fail if you remove the MsgBox command.

Danp2 commented 1 year ago

This is the error shown by geckodriver --

1670774034164   webdriver::server       DEBUG   -> POST /session/36eeb4b9-1697-404a-88de-3023b5a8a8cd/element/1d82eb50-10c6-42b0-ad6a-afab4ffc9127/click {"id":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127"}
1670774034170   webdriver::server       DEBUG   <- 400 Bad Request {"value":{"error":"element not interactable","message":"Element <input id=\"form_20_22\" type=\"submit\"> could not be scrolled into view","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nElementNotInteractableError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:292:5\nwebdriverClickElement@chrome://remote/content/marionette/interaction.sys.mjs:150:11\ninteraction.clickElement@chrome://remote/content/marionette/interaction.sys.mjs:119:11\nclickElement@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:198:29\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:86:31\n"}}

_WD_ElementActionEx yields a different error --

1670774034148   webdriver::server       DEBUG   -> POST /session/36eeb4b9-1697-404a-88de-3023b5a8a8cd/actions {"actions":[{"id":"hover","type":"pointer","parameters":{"pointerType":"mouse"},"actions":[{"type":"pointerMove","duration":100,"x":0,"y":0,"origin":{"ELEMENT":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127","element-6066-11e4-a52e-4f735466cecf":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}]}]}
1670774034157   webdriver::server       DEBUG   <- 500 Internal Server Error {"value":{"error":"move target out of bounds","message":"(42, 992) is out of bounds of viewport width (1282) and height (982)","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nMoveTargetOutOfBoundsError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:372:5\nassertInViewPort@chrome://remote/content/marionette/action.sys.mjs:2107:11\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1011:21\ndispatch/pendingEvents<@chrome://remote/content/marionette/action.sys.mjs:1827:14\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1826:39\ndispatch/chainEvents<@chrome://remote/content/marionette/action.sys.mjs:1753:27\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1755:7\nperformActions@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:464:23\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:139:31\n"}}

A quick web search give numerous instances where others have encountered the same issue while using other tools like Selenium, which confirms my belief that this isn't something that needs to be fixed in this project.

You could try clicking the button via _WD_ExecuteScript as a work around.

mlipok commented 1 year ago

Let me 2 days to analyze this.

demdante commented 1 year ago

This is the error shown by geckodriver --

1670774034164   webdriver::server       DEBUG   -> POST /session/36eeb4b9-1697-404a-88de-3023b5a8a8cd/element/1d82eb50-10c6-42b0-ad6a-afab4ffc9127/click {"id":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127"}
1670774034170   webdriver::server       DEBUG   <- 400 Bad Request {"value":{"error":"element not interactable","message":"Element <input id=\"form_20_22\" type=\"submit\"> could not be scrolled into view","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nElementNotInteractableError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:292:5\nwebdriverClickElement@chrome://remote/content/marionette/interaction.sys.mjs:150:11\ninteraction.clickElement@chrome://remote/content/marionette/interaction.sys.mjs:119:11\nclickElement@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:198:29\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:86:31\n"}}

_WD_ElementActionEx yields a different error --

1670774034148   webdriver::server       DEBUG   -> POST /session/36eeb4b9-1697-404a-88de-3023b5a8a8cd/actions {"actions":[{"id":"hover","type":"pointer","parameters":{"pointerType":"mouse"},"actions":[{"type":"pointerMove","duration":100,"x":0,"y":0,"origin":{"ELEMENT":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127","element-6066-11e4-a52e-4f735466cecf":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}]}]}
1670774034157   webdriver::server       DEBUG   <- 500 Internal Server Error {"value":{"error":"move target out of bounds","message":"(42, 992) is out of bounds of viewport width (1282) and height (982)","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nMoveTargetOutOfBoundsError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:372:5\nassertInViewPort@chrome://remote/content/marionette/action.sys.mjs:2107:11\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1011:21\ndispatch/pendingEvents<@chrome://remote/content/marionette/action.sys.mjs:1827:14\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1826:39\ndispatch/chainEvents<@chrome://remote/content/marionette/action.sys.mjs:1753:27\ndispatch@chrome://remote/content/marionette/action.sys.mjs:1755:7\nperformActions@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:464:23\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:139:31\n"}}

A quick web search give numerous instances where others have encountered the same issue while using other tools like Selenium, which confirms my belief that this isn't something that needs to be fixed in this project.

You could try clicking the button via _WD_ExecuteScript as a work around.

@Danp2 Thanks for looking into this. However, as I said in the original post, the only reason that I know the reason that the action fails is due to another web browser automation tool that I use which is able to perform the automation the first time. The project is open source, and I noticed that the product ensures the element that it wants to perform an action against is scrolled into view before performing the action on it. You can replicate it yourself by performing the following steps:

1) In firefox, navigate to https://addons.mozilla.org/en-US/firefox/ 2) Search for, download, and install the "UI.Vision RPA" plugin for firefox. 3) Launch the new plugin by clicking the icon and you should see a new window open. 4) Create a new macro 5) Once you have created a new macro, click the "Source View (JSON)" tab, and replace all of the text in the input element with the following:

{
  "Name": "hhh",
  "CreationDate": "2022-12-14",
  "Commands": [
    {
      "Command": "bringBrowserToForeground",
      "Target": "",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "open",
      "Target": "https://storeurll.z13.web.core.windows.net/wh.html",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "waitForPageToLoad",
      "Target": "4000",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "select",
      "Target": "id=form_20_Subject",
      "Value": "value=Contact the CFO",
      "Targets": [
        "id=form_20_Subject",
        "name=Subject",
        "xpath=//*[@id=\"form_20_Subject\"]",
        "xpath=//select[@id='form_20_Subject']",
        "xpath=//select",
        "css=#form_20_Subject"
      ],
      "Description": ""
    },
    {
      "Command": "select",
      "Target": "id=form_20_Prefix",
      "Value": "label=Mr.",
      "Targets": [
        "id=form_20_Prefix",
        "name=Prefix",
        "xpath=//*[@id=\"form_20_Prefix\"]",
        "xpath=//select[@id='form_20_Prefix']",
        "xpath=//p[2]/select",
        "css=#form_20_Prefix"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_FirstName",
      "Value": "FirstName",
      "Targets": [
        "id=form_20_FirstName",
        "name=FirstName",
        "xpath=//*[@id=\"form_20_FirstName\"]",
        "xpath=//input[@id='form_20_FirstName']",
        "xpath=//input",
        "css=#form_20_FirstName"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_middle_name",
      "Value": "MiddleNAme",
      "Targets": [
        "id=form_20_middle_name",
        "name=middle_name",
        "xpath=//*[@id=\"form_20_middle_name\"]",
        "xpath=//input[@id='form_20_middle_name']",
        "xpath=//p[4]/input",
        "css=#form_20_middle_name"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_lastName",
      "Value": "LastNAme",
      "Targets": [
        "id=form_20_lastName",
        "name=lastName",
        "xpath=//*[@id=\"form_20_lastName\"]",
        "xpath=//input[@id='form_20_lastName']",
        "xpath=//p[5]/input",
        "css=#form_20_lastName"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_email_address",
      "Value": "fakeemail@fake.com",
      "Targets": [
        "id=form_20_email_address",
        "name=email_address",
        "xpath=//*[@id=\"form_20_email_address\"]",
        "xpath=//input[@id='form_20_email_address']",
        "xpath=//p[9]/input",
        "css=#form_20_email_address"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_phone",
      "Value": "5555555555",
      "Targets": [
        "id=form_20_phone",
        "name=phone",
        "xpath=//*[@id=\"form_20_phone\"]",
        "xpath=//input[@id='form_20_phone']",
        "xpath=//p[10]/input",
        "css=#form_20_phone"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_Street",
      "Value": "809 Fake street",
      "Targets": [
        "id=form_20_Street",
        "name=Street",
        "xpath=//*[@id=\"form_20_Street\"]",
        "xpath=//textarea[@id='form_20_Street']",
        "xpath=//textarea",
        "css=#form_20_Street"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_City",
      "Value": "Fakecity",
      "Targets": [
        "id=form_20_City",
        "name=City",
        "xpath=//*[@id=\"form_20_City\"]",
        "xpath=//input[@id='form_20_City']",
        "xpath=//p[13]/input",
        "css=#form_20_City"
      ],
      "Description": ""
    },
    {
      "Command": "select",
      "Target": "id=form_20_State",
      "Value": "label=Kansas",
      "Targets": [
        "id=form_20_State",
        "name=State",
        "xpath=//*[@id=\"form_20_State\"]",
        "xpath=//select[@id='form_20_State']",
        "xpath=//p[14]/select",
        "css=#form_20_State"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_zipcode",
      "Value": "36898",
      "Targets": [
        "id=form_20_zipcode",
        "name=zipcode",
        "xpath=//*[@id=\"form_20_zipcode\"]",
        "xpath=//input[@id='form_20_zipcode']",
        "xpath=//p[15]/input",
        "css=#form_20_zipcode"
      ],
      "Description": ""
    },
    {
      "Command": "type",
      "Target": "id=form_20_message",
      "Value": "This is a test message,  When the form is submitted using this tool, it will work the first time.",
      "Targets": [
        "id=form_20_message",
        "name=message",
        "xpath=//*[@id=\"form_20_message\"]",
        "xpath=//textarea[@id='form_20_message']",
        "xpath=//p[16]/textarea",
        "css=#form_20_message"
      ],
      "Description": ""
    },
    {
      "Command": "clickAndWait",
      "Target": "id=form_20_22",
      "Value": "",
      "Targets": [
        "id=form_20_22",
        "xpath=//*[@id=\"form_20_22\"]",
        "xpath=//input[@id='form_20_22']",
        "xpath=//p[17]/input",
        "css=#form_20_22"
      ],
      "Description": ""
    }
  ]
}

6) Then Simply click the "play macro" button, and this tool will successfully execute the exact same automation that the au3webdriver udf failed to accomplish on the first try.

I mention this project because I figured it would help resolve the problem, as you could just replicate the behavior that keeps the active element in view at all times in the au3webdriver udf. It might help to compare the differences in the way the browser behavior (eg when the browser scrolls) between the au3webdriver udf method and the method described above. Eg. The above automation method keeps the element that is being acted on in the middle of the browser window at all times, while the au3webdriver udf only scrolls the browser window once the element it needs to interact with is off screen, and then scrolls only so far as to make that one element visible.

Danp2 commented 1 year ago

@demdante

I mention this project because I figured it would help resolve the problem, as you could just replicate the behavior that keeps the active element in view at all times in the au3webdriver udf. It might help to compare the differences in the way the browser

That project doesn't appear to use Webdriver AFAICS, so it appears to me that you are comparing apples to oranges. 🤔

FWIW, the functions in wd_core are intended to implement the Webdriver spec functionality without enhancing or modifying the default behavior. Thus _WD_ElementAction is performing as intended in this case.

The functions in wd_helper add functionality (things like _WD_LinkClickByText and _WD_WaitElement) that isn't present in the W3C specs. As I mentioned earlier, there's _WD_ElementActionEx, which offers an alternative to the standard click functionality in _WD_ElementAction, and it's possible that it could be enhanced to address this issue.

However, I think it is important to understand why the 2nd click only succeeds if the MsgBox was displayed.

Danp2 commented 1 year ago

FWIW, it appears that manually scrolling the window before attempting the click will allow the first click to succeed --

    _WD_ExecuteScript( $sSession, "window.scrollTo(0, document.body.scrollHeight)");
    Sleep(500)
    $theclick = _WD_ElementAction ( $sSession, $sub, "CLICK" )

Edit: Scrolling to the desired element also works --

_WD_ExecuteScript($sSession, "arguments[0].scrollIntoView(false);", '{"' & $_WD_ELEMENT_ID & '":"' & $sub & '"}')
Sleep(500)
$theclick = _WD_ElementAction ( $sSession, $sub, "CLICK" )

Note, neither method works without the half second sleep, so this seems like a timing issue to me.

mlipok commented 1 year ago

Work in progress

mlipok commented 1 year ago

I think that webdriver implementation is not waiting if isScrolledIntoView so I create my own function _WD_scrollIntoView

#include <Array.au3>
#include <File.au3>
#include <Inet.au3>

#include "wd_helper.au3"
#include "wd_capabilities.au3"

Global $sSession

_Main()
Func _Main()
    _WD_Option("Driver", "geckodriver.exe")
    _WD_Option('DriverParams', '--log trace')
    _WD_Option('Port', 4444)
    _WD_CapabilitiesStartup()
    _WD_CapabilitiesAdd('alwaysMatch', 'firefox')
    _WD_CapabilitiesAdd('browserName', 'firefox')
    _WD_CapabilitiesAdd('acceptInsecureCerts', True)
    _WD_CapabilitiesAdd('javascriptEnabled', True)
    _WD_CapabilitiesAdd('acceptInsecureCerts', True)
    Local $sCapabilities = _WD_CapabilitiesGet()
    _WD_Startup()
    If @error Then Exit SetError(2, @error)
    $sSession = _WD_CreateSession($sCapabilities)

    _Example()

    _WD_DeleteSession($sSession)
    _WD_Shutdown()
EndFunc   ;==>_Main

Func _Example()
    _WD_Navigate($sSession, "https://storeurll.z13.web.core.windows.net/wh.html")
    _WD_LoadWait($sSession)

    Local $submit = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_22"]', Default, Default, BitOR($_WD_OPTION_Visible, $_WD_OPTION_Enabled))

    ; try to comment / uncomment the following line
    _WD_scrollIntoView($submit)

    ; This first attempt to click the button will fail, as the command attempts to click the button BEFORE scrolling down the page so that it is visible in the window
    Local $theclick1 = _WD_ElementAction($sSession, $submit, "CLICK")
    MsgBox(1, @ScriptLineNumber, $theclick1)

    Local $theclick2 = _WD_ElementAction($sSession, $submit, "CLICK")
    MsgBox(1, @ScriptLineNumber, $theclick2)
EndFunc   ;==>_Example

Func _WD_scrollIntoView($sElement, $iTimeOut = 1000)
    Local Static $sScript = _
            "var element = arguments[0];" & _
            "element.scrollIntoView(true);"
    _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
    If @error Then Return SetError(@error, @extended, '')

    Local $hTimer = TimerInit()
    Local $bTest
    Do
        __WD_Sleep(100)
        $bTest = __WD_isScrolledIntoView($sElement)
        ConsoleWrite('! ' & $bTest & ' ' & VarGetType($bTest) & @CRLF)
        If $bTest Then ExitLoop
    Until TimerDiff($hTimer) >= $iTimeOut
EndFunc   ;==>_WD_scrollIntoView

Func __WD_isScrolledIntoView($sElement)
    ; https://stackoverflow.com/a/22480938/5314940
    Local Static $sScript = StringReplace( _
            "function isScrolledIntoView(el) {" & _
            "   var rect = el.getBoundingClientRect();" & _
            "   var elemTop = rect.top;" & _
            "   var elemBottom = rect.bottom;" & _
            "   var isVisible = (elemTop >= 0) && (elemBottom <= window.innerHeight);" & _
            "   return isVisible;" & _
            "};" & _
            "var element = arguments[0];" & _
            "return isScrolledIntoView(element);" & _
            "", @TAB, '')
    Local $result = _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
    Return SetError(@error, @extended, $result)
EndFunc   ;==>__WD_isScrolledIntoView

Danp2 commented 1 year ago

@mlipok It seems like the webdriver isn't waiting. And the behavior is the same for each browser I tested. However, clicks work on other websites where the page needs to be scrolls for the target to be in view. Thus I can only guess that there is something "odd" about this particular webpage.

https://stackoverflow.com/a/39914235/5314940

FYI, this doesn't appear to link to the correct topic.

mlipok commented 1 year ago

https://stackoverflow.com/a/39914235/5314940

FYI, this doesn't appear to link to the correct topic.

https://stackoverflow.com/a/488073/5314940 https://stackoverflow.com/a/22480938/5314940

demdante commented 1 year ago

I think that webdriver implementation is not waiting if isScrolledIntoView so I create my own function _WD_scrollIntoView

#include <Array.au3>
#include <File.au3>
#include <Inet.au3>

#include "wd_helper.au3"
#include "wd_capabilities.au3"

Global $sSession

_Main()
Func _Main()
  _WD_Option("Driver", "geckodriver.exe")
  _WD_Option('DriverParams', '--log trace')
  _WD_Option('Port', 4444)
  _WD_CapabilitiesStartup()
  _WD_CapabilitiesAdd('alwaysMatch', 'firefox')
  _WD_CapabilitiesAdd('browserName', 'firefox')
  _WD_CapabilitiesAdd('acceptInsecureCerts', True)
  _WD_CapabilitiesAdd('javascriptEnabled', True)
  _WD_CapabilitiesAdd('acceptInsecureCerts', True)
  Local $sCapabilities = _WD_CapabilitiesGet()
  _WD_Startup()
  If @error Then Exit SetError(2, @error)
  $sSession = _WD_CreateSession($sCapabilities)

  _Example()

  _WD_DeleteSession($sSession)
  _WD_Shutdown()
EndFunc   ;==>_Main

Func _Example()
  _WD_Navigate($sSession, "https://storeurll.z13.web.core.windows.net/wh.html")
  _WD_LoadWait($sSession)

  Local $submit = _WD_WaitElement($sSession, $_WD_LOCATOR_ByXPath, '//*[@id="form_20_22"]', Default, Default, BitOR($_WD_OPTION_Visible, $_WD_OPTION_Enabled))

  ; try to comment / uncomment the following line
  _WD_scrollIntoView($submit)

  ; This first attempt to click the button will fail, as the command attempts to click the button BEFORE scrolling down the page so that it is visible in the window
  Local $theclick1 = _WD_ElementAction($sSession, $submit, "CLICK")
  MsgBox(1, @ScriptLineNumber, $theclick1)

  Local $theclick2 = _WD_ElementAction($sSession, $submit, "CLICK")
  MsgBox(1, @ScriptLineNumber, $theclick2)
EndFunc   ;==>_Example

Func _WD_scrollIntoView($sElement, $iTimeOut = 1000)
  Local Static $sScript = _
          "var element = arguments[0];" & _
          "element.scrollIntoView(true);"
  _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
  If @error Then Return SetError(@error, @extended, '')

  Local $hTimer = TimerInit()
  Local $bTest
  Do
      __WD_Sleep(100)
      $bTest = __WD_isScrolledIntoView($sElement)
      ConsoleWrite('! ' & $bTest & ' ' & VarGetType($bTest) & @CRLF)
      If $bTest Then ExitLoop
  Until TimerDiff($hTimer) >= $iTimeOut
EndFunc   ;==>_WD_scrollIntoView

Func __WD_isScrolledIntoView($sElement)
  ; https://stackoverflow.com/a/22480938/5314940
  Local Static $sScript = StringReplace( _
          "function isScrolledIntoView(el) {" & _
          "   var rect = el.getBoundingClientRect();" & _
          "   var elemTop = rect.top;" & _
          "   var elemBottom = rect.bottom;" & _
          "   var isVisible = (elemTop >= 0) && (elemBottom <= window.innerHeight);" & _
          "   return isVisible;" & _
          "};" & _
          "var element = arguments[0];" & _
          "return isScrolledIntoView(element);" & _
          "", @TAB, '')
  Local $result = _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
  Return SetError(@error, @extended, $result)
EndFunc   ;==>__WD_isScrolledIntoView

Awesome!!! Glad to see this making progress.

mlipok commented 1 year ago

@Danp2 what about implementing the following code to wd_helper.au3:

Func _WD_scrollIntoView($sElement, $iTimeOut = 1000)
    Local Static $sScript = _
            "var element = arguments[0];" & _
            "element.scrollIntoView(true);"
    _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
    If @error Then Return SetError(@error, @extended, '')

    Local $hTimer = TimerInit()
    Local $bTest
    Do
        __WD_Sleep(100)
        $bTest = __WD_isScrolledIntoView($sElement)
        ConsoleWrite('! ' & $bTest & ' ' & VarGetType($bTest) & @CRLF)
        If $bTest Then ExitLoop
    Until TimerDiff($hTimer) >= $iTimeOut
EndFunc   ;==>_WD_scrollIntoView

Func __WD_isScrolledIntoView($sElement)
    ; https://stackoverflow.com/a/22480938/5314940
    Local Static $sScript = StringReplace( _
            "function isScrolledIntoView(el) {" & _
            "   var rect = el.getBoundingClientRect();" & _
            "   var elemTop = rect.top;" & _
            "   var elemBottom = rect.bottom;" & _
            "   var isVisible = (elemTop >= 0) && (elemBottom <= window.innerHeight);" & _
            "   return isVisible;" & _
            "};" & _
            "var element = arguments[0];" & _
            "return isScrolledIntoView(element);" & _
            "", @TAB, '')
    Local $result = _WD_ExecuteScript($sSession, $sScript, __WD_JsonElement($sElement), Default, $_WD_JSON_Value)
    Return SetError(@error, @extended, $result)
EndFunc   ;==>__WD_isScrolledIntoView

mlipok commented 1 year ago

This is the error shown by geckodriver --

1670774034164   webdriver::server       DEBUG   -> POST /session/36eeb4b9-1697-404a-88de-3023b5a8a8cd/element/1d82eb50-10c6-42b0-ad6a-afab4ffc9127/click {"id":"1d82eb50-10c6-42b0-ad6a-afab4ffc9127"}
1670774034170   webdriver::server       DEBUG   <- 400 Bad Request {"value":{"error":"element not interactable","message":"Element <input id=\"form_20_22\" type=\"submit\"> **could not be scrolled into view**","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nElementNotInteractableError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:292:5\nwebdriverClickElement@chrome://remote/content/marionette/interaction.sys.mjs:150:11\ninteraction.clickElement@chrome://remote/content/marionette/interaction.sys.mjs:119:11\nclickElement@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:198:29\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:86:31\n"}}

IMHO: This error with all our tests here, said to us that this is strictly webdriver implementation error.

EDIT: because we are able to use JS to ensure that element is scrolled into the view, but driver is not able and thus return: could not be scrolled into view

mlipok commented 1 year ago

I suppose that webdriver does something like element.scrollIntoView(true); and only onces checks isScrolledIntoView(el).

In my proposed solution we are checking isScrolledIntoView(el) in a loop and waiting till it is scrolled well.

Danp2 commented 1 year ago

what about implementing the following code to wd_helper.au3

I was thinking this functionality could be added to _WD_ElementActionEx. In fact, I plan to review the functionality of _WD_ElementActionEx to see if this needs to be a default action since the current implementation will fail if the element isn't visible in the viewport.

IMHO: This error with all our tests here, said to us that this is strictly webdriver implementation error.

Not sure that I can agree with this. If you review the specs, it says to try to scroll the element's container into view and generate an error if various conditions are met. So it seems that the webdrivers are performing to spec. However, I wonder why the spec calls for scrolling to the element's container instead of the element.

mlipok commented 1 year ago

what about implementing the following code to wd_helper.au3

I was thinking this functionality could be added to _WD_ElementActionEx. In fact, I plan to review the functionality of _WD_ElementActionEx to see if this needs to be a default action since the current implementation will fail if the element isn't visible in the viewport.

Yes we can implement usage of this function into _WD_ElementActionEx() But IMHO _WD_scrollIntoView() should be separate fuction to give an easy way for user to bring element to user eyes.

IMHO: This error with all our tests here, said to us that this is strictly webdriver implementation error.

Not sure that I can agree with this. If you review the specs, it says to try to scroll the element's container into view and generate an error if various conditions are met. So it seems that the webdrivers are performing to spec. However, I wonder why the spec calls for scrolling to the element's container instead of the element.

Agree about the specs. But when I said about implementation error I mean that particular webdriver exe (i.e. geckodriver.exe) implements specs in a wrong way.

Danp2 commented 1 year ago

@mlipok

But IMHO _WD_scrollIntoView() should be separate fuction to give an easy way for user to bring element to user eyes.

Please submit a PR (perhaps after resolving some of the existing ones 😉 )

But when I said about implementation error I mean that particular webdriver exe (i.e. geckodriver.exe) implements specs in a wrong way.

As I mentioned previously, I tested multiple webdrivers and then all exhibited the same behavior. I believe that this is more of a fringe case where the default behavior just doesn't work.

Danp2 / au3WebDriver