Switch Window seems to fail randomly even if it report PASS

sbengo commented 4 years ago

Hi guys,

We have started to use RF to automate web test and retrieve metrics from navigations. Everything has been OK until we have to automate some IE9-prepared websites.

The website open several dialogs so we have to call Switch Window to change window context and continue the test.

We are using the following KWs to force (with 2 retries) a Switch Window that it has been correctly opened, but, even if it reports as PASS, the window is not always switched, so the test fails

Robot

*** Test Cases ***
...

    ${status}   ${value} =  Run Keyword And Ignore Error        Open Dialog
    Run Keyword If  '${status}' == 'FAIL'   Run Keyword And Ignore Error   Open Dialog

    ${win_list}     Get Window Handles
    ${dialog_id}   Get From List   ${win_list}    2
    Switch Window   ${dialog_id}

....

*** Keywords ***
Open Dialog
    Click Element   xpath=(.//td[@id='myLocator'])
    Switch Window   NEW     10s

Logs

On the following gist you can find a real execution and how switch window report as pass but the window context is not really changed:

https://gist.github.com/sbengo/291164d42b68b69c42b5b7fdefbabded

Failed Open Dialog (x2 times) on Switch Window KW
New Window is being retrived, so it exists.
The Switch Window report as PASS but it really doesn't switch window context

So, there is another way to make a secure Switch Window?

Environment

Browser: IE 11 with compatibility mode (website prepared for IE9) Browser driver: Remote Selenium Webdriver IE 32 bits

Libraries

Robot Framework: 3.1.1
SeleniumLibrary: 4.1.0

aaltat commented 4 years ago

From a SeleniumLibrary point of view, everything works. And log doesn't provide any useful information why the failing happens. It was not very clear from the issue, is this a constant problem or is a random problem?

sbengo commented 4 years ago

This is a random problem and I coudln't retrive any common casuistic that can help us to know what is happening.

As we have seen, it is happening more than 50% of the times that is trying to open a Window. On some navigations that we have to interact with more than 5 dialogs, it never ends successfully.

aaltat commented 4 years ago

IE is famous about it being hard to automate and also IE is really slow. This sounds like timing problem, sometimes there is enough delay (selecting windows works) and sometimes there is not (selecting windows does not work). Usually the best advice from me is to not test in IE, but it sounds that it is must requirement for you and therefore it is not option for you.

But how to solve the problem. Selenium and therefore SeleniumLibrary does not know when windows is ready to be selected, it just blindly changes the handle for the window. Therefore I would implement some sort of retry logic and perhaps would timeout after suitable time. Example:

0) Wait that new windows is visible for Selenium. 1) Select correct correct window. 2) Try to find something that should be present in the window. 3) Repeat if failing. Timeout after suitable amount time.

I do not have silver bulled how that should be done for IE, because I use Linux and Chrome with SeleniumLibrary. But because logic is complex (it may example require trying to open the new window multiple times), I would implement this logic in the Python side. There is multiple ways to extend SeleniumLibrary , for that please see the extending documentation.

But because this is turning more of support request and not an bug report (it could be a bug if it fails constantly), I am going to close this issue. For more support please see support section. The Slack and User Group mailing list are most likely your best options.

sbengo commented 4 years ago

Hi @aaltat , thanks for the answer.

First of all, sorry for continiuing with the issue, but I have been doing some test and wanted to share with you.

The output below corresponds to the execution of a Test Case that opens a Window {win_1} from {win_0} and then, opens another window {win_2} from {win_1}: {win_0} -> click -> {win_1} -> click -> {win_2}.

As I have seen, the Get Window Handles request return the ids into array on an arbitrary way. Looking for that, I found that is documented on here: https://w3c.github.io/webdriver/#get-window-handles

So:

Does the KW Switch Window take into account the order or rearrange in some way?
The KW Switch Window | NEW open the last id of the array retrieved by the Window Handle and. Could be not really the last opened window?

Following with the Test Case:

- +--- START KW: SeleniumLibrary.Get Window Handles [ ]
    - GET http://X.X.X.X:4444/session/f6c8f10a-b915-4f9c-bad5-5cc98d5b89b8/window/handles {}
    - Starting new HTTP connection (1): X.X.X.X:4444
    - http://X.X.X.X:4444 "GET /session/f6c8f10a-b915-4f9c-bad5-5cc98d5b89b8/window/handles HTTP/1.1" 200 106
    - Finished Request
    - ${dialog_list1} = ['098060d0-905d-491c-a794-6d47e65eb591', '2f8633c3-0c0e-4dba-9cec-16ef30474627']
- +--- END KW: SeleniumLibrary.Get Window Handles (367)
******
{win_0} = '098060d0-905d-491c-a794-6d47e65eb591'
{win_1} ='2f8633c3-0c0e-4dba-9cec-16ef30474627'
....

- +--- START KW: SeleniumLibrary.Get Window Handles [ ]
    - GET http://X.X.X.X:4444/session/f6c8f10a-b915-4f9c-bad5-5cc98d5b89b8/window/handles {}
    - Starting new HTTP connection (1): X.X.X.X:4444
    - http://X.X.X.X:4444 "GET /session/f6c8f10a-b915-4f9c-bad5-5cc98d5b89b8/window/handles HTTP/1.1" 200 148
    - Finished Request
    - ${dialog_list_2} = ['098060d0-905d-491c-a794-6d47e65eb591', 'b80ea416-d2cf-4319-9680-a77672d7b7a7', '2f8633c3-0c0e-4dba-9cec-16ef30474627']
- +--- END KW: SeleniumLibrary.Get Window Handles (534)
******
{win_0} = '098060d0-905d-491c-a794-6d47e65eb591'
{win_1} !='2f8633c3-0c0e-4dba-9cec-16ef30474627'
 --> {win_1} = 'b80ea416-d2cf-4319-9680-a77672d7b7a7
{win_2} = '2f8633c3-0c0e-4dba-9cec-16ef30474627'

Thanks again, Regards!

aaltat commented 4 years ago

No worries, thank you for not giving up.

SeleniumLibrary does not take in account the order, it will blindly select the last window. It seems that some browsers, like Chrome and Firefox do not obey the specification and provide window handles in ordered list.

To solve your problem, you could use the exclude functionality in the keyword and that should select you the correct window.

But because keyword does not work as documented, I am considering this a bug. It could be hard one fix correctly. but at least we could document the NEW argument value does not work as expected if the browser follows the specification. Another option could be deprecate the NEW argument value and remove it eventually. Need to think what would be correct way solve the problem.

sbengo commented 4 years ago

Hi again @aaltat , thanks for the explanation!

As you have proposed, we will try to use the exclude functionality!

Thanks again, Regards!

robotframework / SeleniumLibrary