Firefox 53 / 56 / 57 Issues

[vladikoff]

My default Firefox update to 53 and now the Intern tests are failing to start properly:

Suite firefox on any platform FAILED SessionNotCreatedException: [POST http://localhost:4444/wd/hub/session / {"desiredCapabilities":{"firefox_profile":"UEsDBBQACAAIAAqylEoAAAAAAAAAAAAAAAAHAAAAdXNlci5qc61XXW/bNhR9368w/DC0wEzLSYNtCYqhS9shQIsA84LtjaClK4s1RWokZdn79TvURyLbsh0DRfKgKPfyfvDcc49KR5YXltI3Y1EUrCwS4YmJ0pvxT6NUKEdv734oB41Ii4WiZNhuYU0FL5aYSisjEpYLLZZ44TJT/Z2RnnthvdTL0+6fnr58YGZN1sqEYOptuZdQF6i2vGYijqnwdVbHbZXUK2YK0pw2nqwWCkdf7Rbanftiq6mqpEZBx41NmsI+JDrQuu5EJ1Kqn1E9e1UTHQkbZ+3lDB8eCtFOGu3YQpl4paTzrz3c1X7eWGKWXJkTT63JeWyFy4ajPZeSkVIszihefaRUlMr/3tz7aTcvFo5VwupHfa+MO9OvnvUj7uz00S7AqixYAbDBMhq+1J0ryIVCLnS6W925mckpnM0rUjGeeWkDeMZiYUp/u1BCr8a7MRNae2OUYwCxsTFuyKh+sEOYJiZniXRhuniN0qS0wAoPczRcfe/2w+h+bJznMbwdPGZ7beiZK7Nc7uLwMJ+eecsPJ1F7aK6Nl+n2CXwwnL4mXxm7ajli0k7RBD33Zch/YJg6l8z7An6bCRqrKfZhAiYF2QmCgTYGit/xLDLpsu0kEFwhnJso0ksfQH91c7N7jV1SYEnHhFKm4ostTxrUw+GwbQWGqPCOAb88N4lQZxBGMa7Zb1kqFeGJGStxNbwwSsZbRLjeTcjt2ztvZez5vttA855dwwxy0uDAgK/69ZFhPOlSkzo3Or7MuSKxQl2vTzA4XBZLkVhfVtqux2XRXLnIpedSv76Ta0lYKkuey82xVbrb+h2Hs+nJpTYarJ5TviA7r/88MlGBpFbSs3Y+un3dEtEgwjsXTwoRvN2eRviheTMideF76/fQ1tI3zPeRzf5NrIWLrSw89npNAnVrHvR9w7eD6XdbLPBts9lZUuZFr4jDoa5okVgJZuGN0uClhpFDXjwm60NvTzk1xMm18DiD0xqzd84FtBRW8ksciRc1sx0GCpWADHnTCm5Lzb3Mw1Re762AZ1DVwONgT+gH32gHlFan176E9+CI2rWMyTG31THDGmFgRtIJJF4gsPDmUc9LyDE3XCEEnfY1323QSoP6wKulh8yxEiHHgdlvp9PZ1c8sws/s9tcoiqbr2d56HToGeDdQsSnWLfGTZ15H19H0xe63sDVQ1ftQ1I91/Rv/Pt2A593Km4JfEt5h2qQ+Wk4durH57mEhgI518TlsWXyfsI580PHuTMTWau/22n63KPJmhQ+DenE//fnQA8GeV+9LxQFr2g9DNKdECqaxAZYC6pZBFuQYHkiETl4FOXU4RWGyYbaQKsCzYYlK+iyGSgyhDj2WpMliwQcdAXGIQep9s4y/mv+kUmJ6w6LRm68iltobl92NHjBzaoQXo8f56B+oNDaL7kZ2ffsOgH87+gOq2kyvolmE39nos8RXmtlMwz9Hnzcf/iKwjp3OWLTXnZ7+qpX5PWQI+KYt5x3GabhdZ/x+OebX0Wit0/ujN2nYbrBjmwJb0kNHsTAC0Hn48sCe+rdElUdE7otLlUl0LnzhnPbYRVf4ym2BdvlRp5uj5TLzKoi0AaakDYAHFq4JrmbWT89vznuckdovR7uyKIxtFuSZLLDbZQr4dCO7B6BOIdcqd9JLv5ft/1BLBwi7/Tsi2gQAADgQAABQSwECLQMUAAgACAAKspRKu/07ItoEAAA4EAAABwAAAAAAAAAAACAApIEAAAAAdXNlci5qc1BLBQYAAAAAAQABADUAAAAPBQAAAAA=","name":"tests/intern_functional","idle-timeout":60,"browserName":"firefox","marionette":true}}] Unable to create new remote session. desired capabilities = Capabilities [{marionette=true, name=tests/intern_functional, browserName=firefox, idle-timeout=60, moz:firefoxOptions=org.openqa.selenium.firefox.FirefoxOptions@392f6c41, firefox_profile=org.openqa.selenium.firefox.FirefoxProfile@7c2df500}], required capabilities = Capabilities [{}] Build info: version: '3.1.0', revision: '86a5d70', time: '2017-02-16 07:57:44 -0800' System info: host: 'Vlads-MBP', ip: '192.168.1.144', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.4', java.version: '1.8.0_111' Driver info: driver.version: FirefoxDriver at Server.createSession <node_modules/intern/node_modules/leadfoot/Server.js:355:15> at <node_modules/intern/lib/executors/Runner.js:198:23> at retry <node_modules/intern/lib/util.js:818:13> at <node_modules/intern/node_modules/dojo/Promise.ts:393:15> at runCallbacks <node_modules/intern/node_modules/dojo/Promise.ts:11:11> at <node_modules/intern/node_modules/dojo/Promise.ts:317:4> at run <node_modules/intern/node_modules/dojo/Promise.ts:237:7> at <node_modules/intern/node_modules/dojo/nextTick.ts:44:3> at nextTickCallbackWith0Args at process._tickCallback TOTAL: tested 0 platforms, 0/0 tests failed; fatal error occurred

Does anyone have this working? I'm trying to find working tunnelOptions. This worked in FF50 to FF52

    tunnel: 'SeleniumTunnel',
    tunnelOptions: {
      // this tells SeleniumTunnel to download geckodriver
      drivers: [{
        name: 'firefox',
        version: '0.11.1'
      }],
      // version of Selenium
      version: '3.1.0'
    }

[vladikoff]

If I update to Selenium 3.3.1 and Geckodriver 0.15.0 I get:

Error: [POST http://localhost:4444/wd/hub/session/611d26b4-6217-4eaa-bbbe-98e5f0af8559/element/28/value / {"value":["s","i","g","n","i","n","0",".","9","4","7","7","3","6","1","0","7","2","7","8","6","1","5","2","@","r","e","s","t","m","a","i","l",".","n","e","t"]}] Expected [object Undefined] undefined to be a string
Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
System info: host: 'Vlads-MBP', ip: '192.168.1.144', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.4', java.version: '1.8.0_111'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Capabilities [{moz:profile=/var/folders/p7/39d7lczd60q26jjqg29z57w00000gn/T/rust_mozprofile.ZdZtPYQsPWNT, rotatable=false, timeouts={implicit=0, pageLoad=300000, script=30000}, pageLoadStrategy=normal, platform=ANY, specificationLevel=0, moz:accessibilityChecks=false, acceptInsecureCerts=false, browserVersion=53.0, platformVersion=16.5.0, moz:processID=46688, browserName=firefox, platformName=darwin}]
Session ID: d4d1378c-9ee2-454a-a0d4-472d0a7ee4dc

There might be some changes in the Spec for send keys. @jason0x43 Do you know if there is a way to get access to marionette logs while the Intern tests are running? I can never figure that one out :(

[vladikoff]

Ref: https://github.com/mozilla/geckodriver/issues/659#issuecomment-295727985

[vladikoff]

After the geckodriver issue addressed, there is a problem in feature detection

if (!('dynamicViewport' in capabilities)) {
  testedCapabilities.dynamicViewport = session.getWindowSize().then(function (originalSize) {
   return session.setWindowSize(originalSize.width, originalSize.height);
  }).then(supported, unsupported);
}

https://github.com/theintern/leadfoot/blob/4f11f10b5a4d429cc3eb5da74fc31a31d0da27a7/src/Server.ts#L536

Commenting this out gets the tests working...

[jason0x43]

Gah, it's annoying when feature tests break FF.

Ok, it looks like the issue is that geckodriver hangs when Leadfoot tries to do a window/x/size call. The W3C standard (and geckodriver 0.16) use a window/x/rect command instead, so in the long run we'll need to update Leadfoot to handle that (https://github.com/theintern/leadfoot/issues/108).

In the short run I'll push out an update to disable that check in FF 53+.

[vladikoff]

@jason0x43 meanwhile, I was wondering how do you get access to Marionette logs via Leadfoot? This way I can file better bug reports or forward those logs to the Selenium team

[jason0x43]

In theory, you'd pass verbose: true as a tunnelOption for SeleniumTunnel, which will cause Selenium to output debug logs to stdout. Unfortunately, there appear to be a couple of problems with that at the moment. Another issue for the day... https://github.com/theintern/digdug/issues/43

[jason0x43]

The FF 53 issues are fixed with Leadfoot 1.7.2 (at least, the get/set window size function doesn't hang). The verbose flag is more functional in Dig Dug 1.6.4.

[vladikoff]

@jason0x43 I'm trying to debug this a bit. If you look at this post:

https://github.com/SeleniumHQ/selenium/issues/4664#issuecomment-328154261

How do we use the new FirefoxOptions class via Intern? Currently I use the config.capabilities.firefox_profile but seems like that's the "legacy way" based on that comment.

[vladikoff]

Ok in case someone finds this the new firefoxOptions seem to work by updating the following:

diff --git a/tests/intern.js b/tests/intern.js
index 67700a0..61b6915 100644
--- a/tests/intern.js
+++ b/tests/intern.js
@@ -78,20 +78,20 @@ function (intern, topic, firefoxProfile) {
       // this tells SeleniumTunnel to download geckodriver
       drivers: [{
         name: 'firefox',
-        version: '0.16.0'
+        version: '0.19.0'
       }],
       // version of Selenium
-      version: '3.4.0'
+      version: '3.6.0'
     }
   };
-
+  config.capabilities['moz:firefoxOptions'] = {};
   // to create a profile, give it the `config` option.
-  config.capabilities.firefox_profile = firefoxProfile(config); //eslint-disable-line camelcase
+  config.capabilities['moz:firefoxOptions'].profile = firefoxProfile(config); //eslint-disable-line camelcase

   if (args.firefoxBinary) {
-    config.capabilities.firefox_binary = args.firefoxBinary; //eslint-disable-line camelcase
+    config.capabilities['moz:firefoxOptions'].binary = args.firefoxBinary; //eslint-disable-line camelcase
   }

One issue I have is in FF56 I get: Missing 'handle' parameter

Error: [POST http://localhost:4444/wd/hub/session/89f3db64-29bf-1f4c-8e24-33cd4a6ba352/window / {"name":"newwindow"}] Missing 'handle' parameter
Build info: version: '3.6.0', revision: '6fbf3ec767', time: '2017-09-27T16:15:40.131Z'
System info: host: 'vladikoffs-MBP', ip: '192.168.1.41', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.13', java.version: '1.8.0_144'
Driver info: driver.version: unknown
  at Test.signup [as test]  <tests/functional/sync_v3_email_first.js:105:10>
Suite firefox on any platform - Firefox Desktop Sync v3 email first FAILED
Error: [POST http://localhost:4444/wd/hub/session/89f3db64-29bf-1f4c-8e24-33cd4a6ba352/window / {"name":"2147483680"}] Missing 'handle' parameter

[jason0x43]

Interesting. In the source for Selenium's FirefoxDriver it looks like the driver just takes the profile value from moz:firefoxOptions and converts it to a firefox_profile capability, so I'm not sure why that step would be necessary.

[vladikoff]

@jason0x43 any ideas where name and handle get set for

Error: [POST http://localhost:4444/wd/hub/session/ab1cd1c6-3eb7-294e-ac4c-7c209307ff1e/window / {"name":"2147483649"}] Missing 'handle' parameter

With this code:

        .getAllWindowHandles()
        .then((handles) => {
          console.log('handles', handles)
          myHandle = handles[1];
        })
        .switchToWindow('2147483649')

I always get handles [ '2147483649', '2147483672' ] but even if hardcode it it is still missing the handler option.

Some refs: https://github.com/SeleniumHQ/selenium/pull/2718 and https://github.com/SeleniumHQ/selenium/pull/2718

[vladikoff]

Also why are the handles always the same names... 🤦‍♂️

[jason0x43]

That handle parameter for switchToWindow is set in Leadfoot. I actually have a TODO there about the parameter name being changed to handle in the WebDriver spec. At the time everything still used name, but I guess geckodriver has switched over.

[vladikoff]

@jason0x43 ugh my bad, I searched for it but failed to find it :P

[jason0x43]

Also, it looks like moz:firefoxOptions is mentioned in the geckodriver docs: https://github.com/mozilla/geckodriver/blob/master/README.md#firefox-capabilities. I need to mention that in Intern's/Leadfoot's docs...

[vladikoff]

@jason0x43 so besides that handle fix, this scenario doesn't work anymore:

If a window is opened in an .execute, example

function openWindow (url, name) {
  var newWindow = window.open(url, name || 'newwindow');
})

return this.parent.execute(openWindow, [ url, windowName ]);

The call to .switchToWindow('newwindow') will fail with:

NoSuchWindow: [POST http://localhost:4444/wd/hub/session/79fd23b4-8d34-7348-9b44-7ab9a87617a1/window / {"handle":"newwindow"}] Unable to locate window: newwindow

The workaround is

.getAllWindowHandles()
  .then(handles => {
     return this.remote.switchToWindow(handles[1])
  })

[jason0x43]

I spent a while today (technically yesterday) testing various combinations of FF + geckodriver + Selenium + cloud service (see https://github.com/theintern/leadfoot/issues/126). FF is a bit of a mess right now. I'm going to push out another leadfoot update tomorrow morning that will include some fixes to improve FF compatibility, but anything less than FF 55 + gd 0.18.0 + Selenium 3.5.0 has issues with at least window sizing and positioning.

[vladikoff]

We switched the workaround to

--        .switchToWindow('newwindow')
++        .then(switchToWindow(1))

[jason0x43]

Regarding the window switching issue, WebDriver handles are opaque identifiers rather than window names. Unfortunately I'm not sure of a straightforward way to associate names with handles, so you either have to retain the handle to a window you open, or guess it from the list of handles returned by getAllWindowHandles.