search

mozilla / geckodriver

WebDriver for Firefox
https://firefox-source-docs.mozilla.org/testing/geckodriver/
Mozilla Public License 2.0
7.19k stars 1.52k forks source link

Instability of the driver #1128

Closed ghost closed 6 years ago

ghost commented 6 years ago

System

Testcase

First of all, what settings do I have to provide to geckodriver to disable auto-update of the firefox on startup? Since firefox 56.0.2 it would update on startup and fail all the tests because of that.

A website with 3 angular 2 routes : localhost/#/login, localhost/#/home, localhost/#/page Login func is on basic auth (if that matters), and pages contain some unique elements. Also, upon successful login, it redirects automatically to #home

The problem is: for no apparent reason, firefox would be stuck at #login route. I use SharedDriver from https://github.com/cucumber/cucumber-jvm/blob/master/examples/java-webbit-websockets-selenium/ to reuse the browser between the tests. The test goes as follows:

  1. clear cache, cookies, etc
  2. go to #login route
  3. input password, click the button
  4. manually switch the route to #/page (driver.get)
  5. check for some unique elements on the page

Repeat the same test 5 times.

What actually happens is after the first time, firefox would get stuck showing the home page on #/login route in the url box. Since I clear the cache, it should reload the website from scratch.

I should also mention that it did not happen before firefox 56.0.2.

Is it the way firefox processes routes that has changed? I have heard that hash-based routes are not great, and one should use html5 routes, but I do not have that capability yet.

Stacktrace

org.openqa.selenium.TimeoutException: Expected condition failed: waiting for LoginPage$$Lambda$119/881662115@6ab4ba9f (tried for 10 second(s) with 1 SECONDS interval)
    at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:82)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:231)
    at com.boeing.toolbox.test.viewelements.LoginPage.isPageLoaded(LoginPage.java:55)
    at com.boeing.toolbox.test.steps.CommonSteps.user_on_page(CommonSteps.java:48)
    at ?.User "Boeing Mechanic" is viewing "Browse" page(src/test/resources/test.feature:13)
Caused by: org.openqa.selenium.NoSuchElementException: Unable to locate element: #username
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '3.6.0', revision: '6fbf3ec767', time: '2017-09-27T15:28:36.4Z'
System info: host: 'host', ip: 'ip', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_144'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{moz:profile=C:\Users\user\AppData\Local\Temp\rust_mozprofile.lJypYuT1laYE, rotatable=false, timeouts={implicit=0, pageLoad=300000, script=30000}, pageLoadStrategy=normal, moz:headless=false, platform=XP, moz:accessibilityChecks=false, acceptInsecureCerts=true, browserVersion=57.0.4, platformVersion=10.0, moz:processID=20872, browserName=firefox, javascriptEnabled=true, platformName=XP, moz:webdriverClick=false}]
Session ID: 82fe6933-c247-4530-b781-f6d6f45cbac4

Trace-level log

https://gist.github.com/oiale/61cbb35e82ba8350a6ef46153f518472

(I am not sure how to use gist correctly, sorry)

andreastt commented 6 years ago

First of all, what settings do I have to provide to geckodriver to disable auto-update of the firefox on startup? Since firefox 56.0.2 it would update on startup and fail all the tests because of that.

First of all it would be helpful if you didn’t coalesce a bunch of different issues together when filing a bug.

Unless you’re passing in a custom profile it is surprising to me that Firefox autoupdates. Can you file a separate bug about this?

A website with 3 angular 2 routes : localhost/#/login, localhost/#/home, localhost/#/page Login func is on basic auth (if that matters), and pages contain some unique elements. Also, upon successful login, it redirects automatically to #home

One observation is that basic auth isn’t supported by WebDriver yet. See https://github.com/w3c/webdriver/issues/385.

In any case it would be helpful if you provided a reduced test case that was self-contained (e.g. didn’t involve Angular or any other dependencies, such as Selenium).

The only thing I can see from the trace log is that you navigate to https://localhost:8080/#/page, for which popstate fires and geckodriver returns. You then inject a long and complicated JavaScript that I don’t understand what does. Then you begin what looks to be an expected condition checking the current URL, which then appears to be https://localhost:8080/#/login.

1515700606200   webdriver::server   DEBUG   -> POST /session/82fe6933-c247-4530-b781-f6d6f45cbac4/url {"url":"https://localhost:8080/#/page"}
1515700606200   geckodriver::marionette TRACE   -> 61:[0,95,"get",{"url":"https://localhost:8080/#/page"}]
1515700606211   Marionette  TRACE   0 -> [0,95,"get",{"url":"https://localhost:8080/#/page"}]
1515700606220   Marionette  DEBUG   Received DOM event "popstate" for "https://localhost:8080/#/page"
1515700606220   Marionette  TRACE   0 <- [1,95,null,{}]
1515700606215   geckodriver::marionette TRACE   <- [1,95,null,{}]
1515700606215   webdriver::server   DEBUG   <- 200 OK {"value": {}}
1515700606226   webdriver::server   DEBUG   -> POST /session/82fe6933-c247-4530-b781-f6d6f45cbac4/execute/async {"script":"var callback = arguments[arguments.length - 1];\nvar rootSelector = 'null';\n\nvar getNg1Hooks = function(selector, injectorPlease) {\r\n  function tryEl(el) {\r\n    try {\r\n      if (!injectorPlease && angular.getTestability) {\r\n        var $$testability = angular.getTestability(el);\r\n        if ($$testability) {\r\n          return {$$testability: $$testability};\r\n        }\r\n      } else {\r\n        var $injector = angular.element(el).injector();\r\n        if ($injector) {\r\n          return {$injector: $injector};\r\n        }\r\n      }\r\n    } catch(err) {}\r\n  }\r\n  function trySelector(selector) {\r\n    var els = document.querySelectorAll(selector);\r\n    for (var i = 0; i < els.length; i++) {\r\n      var elHooks = tryEl(els[i]);\r\n      if (elHooks) {\r\n        return elHooks;\r\n      }\r\n    }\r\n  }\r\n\r\n  if (selector) {\r\n    return trySelector(selector);\r\n  } else if (window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__) {\r\n    var $injector = window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__;\r\n    var $$testability = null;\r\n    try {\r\n      $$testability = $injector.get('$$testability');\r\n    } catch (e) {}\r\n    return {$injector: $injector, $$testability: $$testability};\r\n  } else {\r\n    return tryEl(document.body) ||\r\n        trySelector('[ng-app]') || trySelector('[ng\\\\:app]') ||\r\n        trySelector('[ng-controller]') || trySelector('[ng\\\\:controller]');\r\n  }\r\n}\n\r\n\r\n  try {\r\n    // Wait for both angular1 testability and angular2 testability.\r\n\r\n    var testCallback = callback;\r\n\r\n    // Wait for angular1 testability first and run waitForAngular2 as a callback\r\n    var waitForAngular1 = function(callback) {\r\n\r\n      if (window.angular) {\r\n        var hooks = getNg1Hooks(rootSelector);\r\n        if (!hooks){\r\n          callback();  // not an angular1 app\r\n        }\r\n        else{\r\n          if (hooks.$$testability) {\r\n            hooks.$$testability.whenStable(callback);\r\n          } else if (hooks.$injector) {\r\n            hooks.$injector.get('$browser')\r\n                .notifyWhenNoOutstandingRequests(callback);\r\n          } else if (!!rootSelector) {\r\n            throw new Error(\r\n                'Could not automatically find injector on page: \"' +\r\n                window.location.toString() + '\".  Consider using config.rootEl');\r\n          } else {\r\n            throw new Error(\r\n                'root element (' + rootSelector + ') has no injector.' +\r\n                ' this may mean it is not inside ng-app.');\r\n          }\r\n        }\r\n      }\r\n      else {callback();}  // not an angular1 app\r\n    };\r\n\r\n    // Wait for Angular2 testability and then run test callback\r\n    var waitForAngular2 = function() {\r\n      if (window.getAngularTestability) {\r\n        if (rootSelector) {\r\n          var testability = null;\r\n          var el = document.querySelector(rootSelector);\r\n          try{\r\n            testability = window.getAngularTestability(el);\r\n          }\r\n          catch(e){}\r\n          if (testability) {\r\n            return testability.whenStable(function() { testCallback(); });\r\n          }\r\n        }\r\n\r\n        // Didn't specify root element or testability could not be found\r\n        // by rootSelector. This may happen in a hybrid app, which could have\r\n        // more than one root.\r\n        var testabilities = window.getAllAngularTestabilities();\r\n        var count = testabilities.length;\r\n\r\n        // No angular2 testability, this happens when\r\n        // going to a hybrid page and going back to a pure angular1 page\r\n        if (count === 0) {\r\n          return testCallback();\r\n        }\r\n\r\n        var decrement = function() {\r\n          count--;\r\n          if (count === 0) {\r\n            testCallback();\r\n          }\r\n        };\r\n        testabilities.forEach(function(testability) {\r\n          testability.whenStable(decrement);\r\n        });\r\n\r\n      }\r\n      else {testCallback();}  // not an angular2 app\r\n    };\r\n\r\n    if (!(window.angular) && !(window.getAngularTestability)) {\r\n      // no testability hook\r\n      throw new Error(\r\n          'both angularJS testability and angular testability are undefined.' +\r\n          '  This could be either ' +\r\n          'because this is a non-angular page or because your test involves ' +\r\n          'client-side navigation, which can interfere with Protractor\\'s ' +\r\n          'bootstrapping.  See http://git.io/v4gXM for details');\r\n    } else {waitForAngular1(waitForAngular2);}  // Wait for angular1 and angular2\r\n                                                // Testability hooks sequentially\r\n\r\n  } catch (err) {\r\n    callback(err.message);\r\n  }\r\n\r\n","args":[]}
1515700606226   geckodriver::marionette TRACE   -> 4945:[0,96,"executeAsyncScript",{"args":[],"newSandbox":false,"script":"var callback = arguments[arguments.length - 1];\nvar rootSelector = 'null';\n\nvar getNg1Hooks = function(selector, injectorPlease) {\r\n  function tryEl(el) {\r\n    try {\r\n      if (!injectorPlease && angular.getTestability) {\r\n        var $$testability = angular.getTestability(el);\r\n        if ($$testability) {\r\n          return {$$testability: $$testability};\r\n        }\r\n      } else {\r\n        var $injector = angular.element(el).injector();\r\n        if ($injector) {\r\n          return {$injector: $injector};\r\n        }\r\n      }\r\n    } catch(err) {}\r\n  }\r\n  function trySelector(selector) {\r\n    var els = document.querySelectorAll(selector);\r\n    for (var i = 0; i < els.length; i++) {\r\n      var elHooks = tryEl(els[i]);\r\n      if (elHooks) {\r\n        return elHooks;\r\n      }\r\n    }\r\n  }\r\n\r\n  if (selector) {\r\n    return trySelector(selector);\r\n  } else if (window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__) {\r\n    var $injector = window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__;\r\n    var $$testability = null;\r\n    try {\r\n      $$testability = $injector.get('$$testability');\r\n    } catch (e) {}\r\n    return {$injector: $injector, $$testability: $$testability};\r\n  } else {\r\n    return tryEl(document.body) ||\r\n        trySelector('[ng-app]') || trySelector('[ng\\\\:app]') ||\r\n        trySelector('[ng-controller]') || trySelector('[ng\\\\:controller]');\r\n  }\r\n}\n\r\n\r\n  try {\r\n    // Wait for both angular1 testability and angular2 testability.\r\n\r\n    var testCallback = callback;\r\n\r\n    // Wait for angular1 testability first and run waitForAngular2 as a callback\r\n    var waitForAngular1 = function(callback) {\r\n\r\n      if (window.angular) {\r\n        var hooks = getNg1Hooks(rootSelector);\r\n        if (!hooks){\r\n          callback();  // not an angular1 app\r\n        }\r\n        else{\r\n          if (hooks.$$testability) {\r\n            hooks.$$testability.whenStable(callback);\r\n          } else if (hooks.$injector) {\r\n            hooks.$injector.get('$browser')\r\n                .notifyWhenNoOutstandingRequests(callback);\r\n          } else if (!!rootSelector) {\r\n            throw new Error(\r\n                'Could not automatically find injector on page: \"' +\r\n                window.location.toString() + '\".  Consider using config.rootEl');\r\n          } else {\r\n            throw new Error(\r\n                'root element (' + rootSelector + ') has no injector.' +\r\n                ' this may mean it is not inside ng-app.');\r\n          }\r\n        }\r\n      }\r\n      else {callback();}  // not an angular1 app\r\n    };\r\n\r\n    // Wait for Angular2 testability and then run test callback\r\n    var waitForAngular2 = function() {\r\n      if (window.getAngularTestability) {\r\n        if (rootSelector) {\r\n          var testability = null;\r\n          var el = document.querySelector(rootSelector);\r\n          try{\r\n            testability = window.getAngularTestability(el);\r\n          }\r\n          catch(e){}\r\n          if (testability) {\r\n            return testability.whenStable(function() { testCallback(); });\r\n          }\r\n        }\r\n\r\n        // Didn't specify root element or testability could not be found\r\n        // by rootSelector. This may happen in a hybrid app, which could have\r\n        // more than one root.\r\n        var testabilities = window.getAllAngularTestabilities();\r\n        var count = testabilities.length;\r\n\r\n        // No angular2 testability, this happens when\r\n        // going to a hybrid page and going back to a pure angular1 page\r\n        if (count === 0) {\r\n          return testCallback();\r\n        }\r\n\r\n        var decrement = function() {\r\n          count--;\r\n          if (count === 0) {\r\n            testCallback();\r\n          }\r\n        };\r\n        testabilities.forEach(function(testability) {\r\n          testability.whenStable(decrement);\r\n        });\r\n\r\n      }\r\n      else {testCallback();}  // not an angular2 app\r\n    };\r\n\r\n    if (!(window.angular) && !(window.getAngularTestability)) {\r\n      // no testability hook\r\n      throw new Error(\r\n          'both angularJS testability and angular testability are undefined.' +\r\n          '  This could be either ' +\r\n          'because this is a non-angular page or because your test involves ' +\r\n          'client-side navigation, which can interfere with Protractor\\'s ' +\r\n          'bootstrapping.  See http://git.io/v4gXM for details');\r\n    } else {waitForAngular1(waitForAngular2);}  // Wait for angular1 and angular2\r\n                                                // Testability hooks sequentially\r\n\r\n  } catch (err) {\r\n    callback(err.message);\r\n  }\r\n\r\n","scriptTimeout":null,"specialPowers":false}]
1515700606228   Marionette  TRACE   0 -> [0,96,"executeAsyncScript",{"args":[],"newSandbox":false,"script":"var callback = arguments[arguments.length - 1];\nvar rootSelector = 'null';\n\nvar getNg1Hooks = function(selector, injectorPlease) {\r\n  function tryEl(el) {\r\n    try {\r\n      if (!injectorPlease && angular.getTestability) {\r\n        var $$testability = angular.getTestability(el);\r\n        if ($$testability) {\r\n          return {$$testability: $$testability};\r\n        }\r\n      } else {\r\n        var $injector = angular.element(el).injector();\r\n        if ($injector) {\r\n          return {$injector: $injector};\r\n        }\r\n      }\r\n    } catch(err) {}\r\n  }\r\n  function trySelector(selector) {\r\n    var els = document.querySelectorAll(selector);\r\n    for (var i = 0; i < els.length; i++) {\r\n      var elHooks = tryEl(els[i]);\r\n      if (elHooks) {\r\n        return elHooks;\r\n      }\r\n    }\r\n  }\r\n\r\n  if (selector) {\r\n    return trySelector(selector);\r\n  } else if (window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__) {\r\n    var $injector = window.__TESTABILITY__NG1_APP_ROOT_INJECTOR__;\r\n    var $$testability = null;\r\n    try {\r\n      $$testability = $injector.get('$$testability');\r\n    } catch (e) {}\r\n    return {$injector: $injector, $$testability: $$testability};\r\n  } else {\r\n    return tryEl(document.body) ||\r\n        trySelector('[ng-app]') || trySelector('[ng\\\\:app]') ||\r\n        trySelector('[ng-controller]') || trySelector('[ng\\\\:controller]');\r\n  }\r\n}\n\r\n\r\n  try {\r\n    // Wait for both angular1 testability and angular2 testability.\r\n\r\n    var testCallback = callback;\r\n\r\n    // Wait for angular1 testability first and run waitForAngular2 as a callback\r\n    var waitForAngular1 = function(callback) {\r\n\r\n      if (window.angular) {\r\n        var hooks = getNg1Hooks(rootSelector);\r\n        if (!hooks){\r\n          callback();  // not an angular1 app\r\n        }\r\n        else{\r\n          if (hooks.$$testability) {\r\n            hooks.$$testability.whenStable(callback);\r\n          } else if (hooks.$injector) {\r\n            hooks.$injector.get('$browser')\r\n                .notifyWhenNoOutstandingRequests(callback);\r\n          } else if (!!rootSelector) {\r\n            throw new Error(\r\n                'Could not automatically find injector on page: \"' +\r\n                window.location.toString() + '\".  Consider using config.rootEl');\r\n          } else {\r\n            throw new Error(\r\n                'root element (' + rootSelector + ') has no injector.' +\r\n                ' this may mean it is not inside ng-app.');\r\n          }\r\n        }\r\n      }\r\n      else {callback();}  // not an angular1 app\r\n    };\r\n\r\n    // Wait for Angular2 testability and then run test callback\r\n    var waitForAngular2 = function() {\r\n      if (window.getAngularTestability) {\r\n        if (rootSelector) {\r\n          var testability = null;\r\n          var el = document.querySelector(rootSelector);\r\n          try{\r\n            testability = window.getAngularTestability(el);\r\n          }\r\n          catch(e){}\r\n          if (testability) {\r\n            return testability.whenStable(function() { testCallback(); });\r\n          }\r\n        }\r\n\r\n        // Didn't specify root element or testability could not be found\r\n        // by rootSelector. This may happen in a hybrid app, which could have\r\n        // more than one root.\r\n        var testabilities = window.getAllAngularTestabilities();\r\n        var count = testabilities.length;\r\n\r\n        // No angular2 testability, this happens when\r\n        // going to a hybrid page and going back to a pure angular1 page\r\n        if (count === 0) {\r\n          return testCallback();\r\n        }\r\n\r\n        var decrement = function() {\r\n          count--;\r\n          if (count === 0) {\r\n            testCallback();\r\n          }\r\n        };\r\n        testabilities.forEach(function(testability) {\r\n          testability.whenStable(decrement);\r\n        });\r\n\r\n      }\r\n      else {testCallback();}  // not an angular2 app\r\n    };\r\n\r\n    if (!(window.angular) && !(window.getAngularTestability)) {\r\n      // no testability hook\r\n      throw new Error(\r\n          'both angularJS testability and angular testability are undefined.' +\r\n          '  This could be either ' +\r\n          'because this is a non-angular page or because your test involves ' +\r\n          'client-side navigation, which can interfere with Protractor\\'s ' +\r\n          'bootstrapping.  See http://git.io/v4gXM for details');\r\n    } else {waitForAngular1(waitForAngular2);}  // Wait for angular1 and angular2\r\n                                                // Testability hooks sequentially\r\n\r\n  } catch (err) {\r\n    callback(err.message);\r\n  }\r\n\r\n","scriptTimeout":null,"specialPowers":false}]
1515700607013   Marionette  TRACE   0 <- [1,96,null,{"value":null}]
1515700607013   geckodriver::marionette TRACE   <- [1,96,null,{"value":null}]
1515700607013 webdriver::server DEBUG <- 200 OK {"value":null}
ghost commented 6 years ago

@andreastt Hello,

Thank you for your feedback. It was very helpful trying to isolate the problem using more basic examples, and I think that in fact it had nothing to do with geckodriver itself.

You then inject a long and complicated JavaScript that I don’t understand what does.

That javascript waits for angular testabilities to complete.

What happened (I think) was that things were going too fast. I was logging in and invoking a different url without waiting for route redirect to finish. I just added a wait for loading the #home route, and now everything works fine.

I am still not sure whether it is the way angular works, the way I made this specific example, or the way hash-based routes work in browsers.

As far as the update problem, I will try to rollback to 56.0.2, and if the problem persists, I will create a new bug thread.

lock[bot] commented 5 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. If you have run into an issue you think is related, please open a new issue.