Chrome 65 support needs to to be released

[rjbeers]

Chrome 65 started rolling out to users' browsers. In order to support Chrome 65, the code already committed to master needs to be released to npm.

[jason0x43]

We're still considering how best to handle webdriver updates (like, just updating the default would break CI tests that use older versions of chrome). In the mean time, you can explicitly use a newer driver with tunnelOptions in your intern config:

"tunnelOptions": {
  "drivers": [ { "name": "chrome", "version": "2.36" } ]
}

[rjbeers]

Thanks for the quick response! I'd recommend not including webdriver updates in your API contract.

If you included webdriver changes in your contract, you would be tied to their releases and this change would be a major version bump. Then changes to the actual code provided would be left to a new major version or backported through major versions. Maintaining support for the latest stable set of browsers would be left to each individual consumer of your package instead of being centrally supported.

If latest stable webdrivers are used without a major bump, you can freely maintain your code and issue major versions for true API changes. If a team is relying on testing a specific set of browsers indefinitely, it would make sense for them to configure their setup to use a specific webdriver version as well. Providing support and testing capabilities for the latest stable browsers should be assumed as default behavior.

As for some evidence to support this rationale, evergreen browsers will always have more usage than older versions (https://www.w3counter.com/globalstats.php) and Chromedriver, for instance, already supports backward compatibility while the vast majority of users are upgrading from the most recent versions.

[jason0x43]

I'd recommend not including webdriver updates in your API contract.

If you mean that we shouldn't need to version bump digdug to make new webdriver versions the default, that would be ideal, but there are a couple of issues with doing that. One is that we don't always have a robust way to determine what the current webdriver version is. For geckodriver we can find the most recent release on github, for chromdriver we can look at https://chromedriver.storage.googleapis.com/index.html, and IE driver tracks the current Selenium version. MicrosoftWebDriver (Edge), however, not only isn't available from a regular endpoint, but it's dependent on the user's operating system version (because Edge isn't standalone).

The other problem we run into is that webdriver versions, including Chromedriver, do not guarantee backward compatibility with older browsers or older versions of Selenium. For example, chromedriver 2.36 claims compatibility with Chrome 64-66; it might work with older versions, but it might not. Simply updating webdrivers silently can break tests; this happened to us a few months ago when updating the default webdriver versions in a minor version bump caused several people's CI tests to stop working. So while changing the webdriver version doesn't change digdug's public API, it can definitely affect digdug's overall functionality, which breaks user's expectations of how a semver-ed package will behave.

If a team is relying on testing a specific set of browsers indefinitely, it would make sense for them to configure their setup to use a specific webdriver version as well. Providing support and testing capabilities for the latest stable browsers should be assumed as default behavior.

This is what we're looking at going forward. Even though updating driver versions might break functionality for older browsers, it will allow the more common case of evergreen browsers to work more of the time. We'll document more clearly that digdug releases will track (roughly) the current versions of the webdrivers it supports and Selenium, and that users needing to test older versions of browsers should pin the driver version to something compatible.

[lethevimlet]

Still having trouble with this, functional test fail on latest chrome, now on webdriver 2.37

(ノಠ益ಠ)ノ彡┻━┻
JavaScriptError: [POST http://localhost:4444/wd/hub/session/31047234bbf994e0c200eaeb9e38d337/execute / {"script":"return (function getCoverageData(coverageVariable) {\n        var coverageData = (function () {\n
          return this;\n        })()[coverageVariable];\n        return coverageData && JSON.stringify(coverageData);\n    }).apply(this, arguments);","args":["__coverage__"]}] unknown error: call function result
missing 'value'
  (Session info: chrome=65.0.3325.181)
  (Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.16299 x86_64)
  at Server.post  <node_modules\src\Server.ts:367:14>
  at runRequest  <node_modules\src\Session.ts:132:33>
  at Task_1.default.cancelled  <node_modules\src\Session.ts:164:38>
  at new Task  <node_modules\@dojo\core\async\Task.ts:239:3>
  at InitializedProxiedSession.Session._delegateToServer  <node_modules\src\Session.ts:110:9>
  at InitializedProxiedSession.Session.serverPost  <node_modules\src\Session.ts:174:14>
  at InitializedProxiedSession.Session.execute  <node_modules\src\Session.ts:368:20>
  at <node_modules\@vimlet\alpha\src\lib\ProxiedSession.ts:143:16>
  at <node_modules\@dojo\core\async\Task.ts:351:12>
  at handler  <node_modules\@dojo\core\async\ExtensiblePromise.ts:229:14>

[miketaylr]

Still having trouble with this, functional test fail on latest chrome, now on webdriver 2.37

Once I manually deleted the chromedriver binary from /node_modules/@theintern/digdug/selenium-standalone, 2.37 worked for me with Chrome locally.

I'm guessing digdug doesn't try to check chromerdiver binary versions and replace them if/when it makes sense?

[jason0x43]

I'm guessing digdug doesn't try to check chromerdiver binary versions and replace them if/when it makes sense?

Only for the Selenium server right now since it's the only one with a versioned filename (like selenium-standalone-3.3.1.jar), but a fix is coming for the other webdrivers.

[rjbeers]

The intern repo now states:

Note that the library Intern uses to manage Selenium and WebDrivers tracks the current versions of those applications; this allows new installs of Intern to work with current browser versions. However, this can also cause Intern to stop being able to drive older versions of browsers. If you need Intern to work with a specific version of a browser, pin your config to specific versions of Selenium and the WebDrivers.

With 66 rolling out shortly, I have to ping asking the status of this repo tracking the latest driver version as it says it will.

[jason0x43]

The next version, which should be released in the next few days, will point to the current Selenium, Chromedriver, etc. The release cadence should pick up in the future since we're more firmly establishing that Dig Dug is going to track current drivers.

[rjbeers]

Cool. Are you just going to be using something like this for the latest version, or are you saying that you will be manually handling this, but with frequent releases?

let {body: latestChromeDriverVersion} = await got('https://chromedriver.storage.googleapis.com/LATEST_RELEASE');

[jason0x43]

Are you just going to be using something like this for the latest version, or are you saying that you will be manually handling this, but with frequent releases?

More frequent releases. There are a couple of potential problems with using 'LATEST_RELEASE': 1) we can't do this for all the drivers, and 2) we need to perform at least minimal testing to ensure the latest release actually works with Dig Dug.

Updated driver versions are now available in release 2.1.0, which is used by Intern 4.2.0.