Mochify does not work on Browserstack with Safari 12 or Firefox 64

[jandockx]

Mochify does not work on Browserstack, with Safari 12 on Mojave and Firefox 64 on Windows 10, High Sierra nor Mojave.

It did work with Safari 11 and Firefox 63 on High Sierra, and Firefox 63 on WIndows 10.

# Firefox - OS X:
POST /wd/hub/session/fb0858d69e0c47c0c71453957267f2560049b6e8/execute
Unexpected HTTP status: 404 Not Found
server: nginx/1.10.1
date: Fri, 04 Jan 2019 16:15:45 GMT
content-type: application/json;charset=utf-8
content-length: 158
connection: keep-alive
expires: Thu, 01 Jan 1970 00:00:00 GMT
cache-control: no-cache, no-cache
Response Status Code: undefined
Response Message:
    POST /session/fb0858d69e0c47c0c71453957267f2560049b6e8/execute did not match a known command

See

• https://travis-ci.org/Toryt/contracts/jobs/475364263
• https://automate.browserstack.com/builds/57601f17429dab8a20183726ff4681b5af2c1194/sessions/a5c25bc365c33545eacd64b59dcdffdfeb7b847b

[jandockx]

Tested with 5.8.0, 5.8.1, and 6.0.4

[jandockx]

Correction: it has worked with Firefox 64, but stopped working between 2018-12-22 and 2018-12-29, on both High Sierra and Windows 10. It is strange that the 'same' issue also happens with Safari 12 / Mojave.

[mantoni]

Thank you for reporting. It is strange that it works in some browsers and doesn’t work in others since the client side code shouldn‘t have changed.

You’re running a lot of test cases there. Can I ask you to retry with a single, simple test case? I had issues before that only occurred when a lot of tests were executed or the tests were blocking for a long time.

[jandockx]

Hi @mantoni ! You're quick!

I'm just trying to get a grip on this, so reporting on the fly:

The logs say that both the last working and the first failing run use

• browserstack.geckodriver: 0.21.0,
• browserstack.selenium.jar.version: 3.11.0

So, there was no switch there either.

In the Selenium logs, there is a difference in the first line. The logs for Windows / Firefox 64 of Travis/651 say

INFO [ActiveSessionFactory.apply] - Capabilities are: Capabilities {64bit: true, acceptSslCert: false, acceptSslCerts: false, browser: firefox, browserName: firefox, browser_version: 64.0, browserstack.appiumLogs: false, browserstack.console: ALL, browserstack.debug: false, browserstack.geckodriver: 0.21.0, browserstack.ie.noFlash: false, browserstack.key: <>, browserstack.selenium.jar.version: 3.11.0, browserstack.seleniumLogs: true, browserstack.user: <>, browserstack.video: true, browserstack.video.disableWaterMark: false, firefox_binary: c:\Program Files (x86)\fire..., firefox_profile: UEsDBBQAAAAIADNwWUmK7I+M9AM..., javascriptEnabled: true, moz:firefoxOptions: {args: [-start-debugger-server, 9222], prefs: {devtools.chrome.enabled: true, devtools.debugger.prompt-connection: false, devtools.debugger.remote-enabled: true}}, orig_os: win10, os: Windows, os_version: 10, proxy_type: node, version: 64.0}

Note:

browser: firefox,
browserName: firefox,
browser_version: 64.0,
…
browserstack.console: ALL,
…
browserstack.accessKey: <>,
browserstack.user: <>,
…
os_version: 10

The logs Travis/654 say

INFO [ActiveSessionFactory.apply] - Capabilities are: Capabilities {64bit: true, acceptSslCert: false, acceptSslCerts: false, browser: firefox, browserName: firefox, browser_name: Firefox, browser_version: 64.0, browserstack.accessKey: <>, browserstack.appiumLogs: false, browserstack.console: ALL, browserstack.consoleLogs: verbose, browserstack.debug: false, browserstack.geckodriver: 0.21.0, browserstack.ie.noFlash: false, browserstack.key: <>, browserstack.selenium.jar.version: 3.11.0, browserstack.seleniumLogs: true, browserstack.user: <>, browserstack.userName: <>, browserstack.video: true, browserstack.video.disableWaterMark: false, buildName: travis/00654, firefox_binary: c:\Program Files (x86)\fire..., firefox_profile: UEsDBBQAAAAIADNwWUmK7I+M9AM..., javascriptEnabled: true, moz:firefoxOptions: {args: [-start-debugger-server, 9222], prefs: {devtools.chrome.enabled: true, devtools.debugger.prompt-connection: false, devtools.debugger.remote-enabled: true}}, orig_os: win10, os: Windows, osVersion: 10, os_version: 10, projectName: Toryt contracts, proxy_type: node, version: 64.0}

Note:

browser: firefox,
browserName: firefox,
browser_name: Firefox,
browser_version: 64.0,
…
browserstack.console: ALL,
browserstack.consoleLogs: verbose,
…
browserstack.accessKey: <>,
browserstack.key: <>,
browserstack.user: <>,
browserstack.userName: <>,
…
buildName: travis/00654,
…
osVersion: 10,
os_version: 10,
projectName: Toryt contracts,

browser_name, browserstack.consoleLogs, browserstack.key, browserstack.userName, buildName, osVersion and projectName are added, although neither mochify (with a package-lock, nor selenium, nor the web-driver nor the Firefox version, nor the OS, changed.

Just observing.

[jandockx]

@mantoni : I'll see what I can do.

[jandockx]

@mantoni Ok, removed all tests except 1. Still the same problem.

Safari 12 on Mojave fails, Safari 11 on High Sierra works. Firefox 64 on Mojave and Windows 10 fails.

See build Travis/00666 (mmh, what a coincidence ;-) )

• https://github.com/Toryt/contracts/tree/feature/firefox-browserstack-problem
• https://travis-ci.org/Toryt/contracts/jobs/475435690
• https://automate.browserstack.com/builds/da25456ea3c713f0fa5c96e6ffb836194ff134b4/sessions/22c2ed18fa67483a0c54929bbbb75e0b7134a10f
• https://automate.browserstack.com/builds/da25456ea3c713f0fa5c96e6ffb836194ff134b4/sessions/90ff804425fddcc9ca71793a0d3a5bddd75d37b3
• https://automate.browserstack.com/builds/da25456ea3c713f0fa5c96e6ffb836194ff134b4/sessions/9bd17d97c9759650bdf7018e24b96b097a81283a

The more I look into this, the more I get the impression that this must be a Browserstack issue. Some update they did between 2018-12-22 and 2018-12-29. I am sorry I inconvenienced you.

[mantoni]

No worries. The issue on the Mochify side is clearly the error reporting. It is kind of difficult to extract meaningful information via WebDriver.

[mantoni]

Ok, I found some time to investigate the issue and I believe that tge cause is a transition to Selenium 3. I guess that for some browsers the default Selenium version has changed to a newer protocol. I'm personally using SauceLabs and ran some tests there.

What I did in node_modules/min-wd/lib/driver.js:

• Line 36: var endpoint = context.asyncPolling ? '/execute/async' : '/execute/sync';
• Line 121: request(context, 'POST', '/execute/sync', {
• Line 138: request(context, 'POST', '/execute/sync', {

On SauceLabs this still works with all browsers I'm testing, except for Safari which only likes the old endpoints. If you have the possibility to run BrowserStack tests from your local machine, I'd appreciate if you could try applying these changes to see if they improve things for you. Thanks.

[jandockx]

@mantoni

As requested:

• check out https://github.com/Toryt/contracts.git, branch origin/feature/firefox-browserstack-problem
• npm cit
• npm run wd
  • needs Browserstack credentials
  • 1 simple Mocha test in each browser,
  • failures observed with FF on Mojave and WIndows 10
  • POST /session/5fe53213aa6ddffca493325647b87c19437472dc/execute did not match a known command
  • failure on Safari on Mojave
  • The command 'POST /session/a42bc4268658dd1a33c07aa508db29771f8a518f/execute' was not found.
• do the 3 changes you described above
• npm run wd
  • FF on Mojave, WIndows 10, and Safari on Mojave ok
  • Chrome on Mojave and Windows 10, and Edge on WIndows 10 ok
  • failures for Safari on High Sierra
  • The command 'POST /session/c1ddd6eeb37ce99b71a4716b182da0ff903b0f4d/execute/sync' was not found.
  • failures for iPhoneX, and for Samsung Galaxy S9 and Samsung Galaxy Note 4
  • The URL '/wd/hub/session/f43e17eae7498b10f4b0e5559ef756be48c89cc6/execute/sync' did not map to a valid resource

[jandockx]

This seems to no longer be an issue.