Firefox stable daily runs are failing with NoSuchWindowException

[stephenmcgruer]

See https://github.com/web-platform-tests/wpt/commits/epochs/daily

Example run: https://community-tc.services.mozilla.com/tasks/groups/X50KGIpZSMq0Ag7vQhbM6A

Example task log: https://community-tc.services.mozilla.com/tasks/T13biD2PSOWC73kPlvymBw/runs/0/logs/https%3A%2F%2Fcommunity-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FT13biD2PSOWC73kPlvymBw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log

 0:13.61 SUITE_START: web-platform-test - running 856 tests
 0:47.36 INFO Running testharness tests
 0:47.39 INFO Setting up ssl
 0:47.44 certutil Full command: /usr/bin/certutil -N -d /tmp/tmprYMnHs.mozrunner -f /tmp/tmprYMnHs.mozrunner/.crtdbpw
certutil 
 0:47.48 certutil 
 0:47.50 certutil 
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

web-platform-tests                                           CT,, 

 0:47.52 INFO Application command: /home/test/build/firefox/firefox --marionette about:blank -profile /tmp/tmpogOO9n
 0:47.53 INFO Application command: /home/test/build/firefox/firefox --marionette about:blank -profile /tmp/tmprbo6jv
 0:47.54 INFO Starting runner
Failed to read: session.screen0.titlebar.left
Setting default value
Failed to read: session.screen0.titlebar.right
Setting default value
 0:49.75 pid:1275 Full command: /home/test/build/firefox/firefox --marionette about:blank -profile /tmp/tmpogOO9n
pid:1275 console.error: SearchCache: "_readCacheFile: Error reading cache file:" (new Error("", "(unknown module)"))
 0:51.05 pid:1275 1596676421960 Marionette  INFO    Listening on port 47423
 0:51.40 TEST_START: /WebCryptoAPI/derive_bits_keys/ecdh_bits.https.any.worker.html
 0:51.42 ERROR Traceback (most recent call last):
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 126, in run_test
    return self.executor.run_test(test)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/base.py", line 285, in run_test
    self.on_environment_change(test.environment)
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 776, in on_environment_change
    self.protocol.testharness.load_runner(new_environment["protocol"])
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 140, in load_runner
    self._close_windows()
  File "/home/test/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 180, in _close_windows
    self.marionette.switch_to_window(runner_handle)
  File "/home/test/web-platform-tests/_venv2/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1360, in switch_to_window
    self._send_message("WebDriver:SwitchToWindow", {"handle": handle, "focus": focus})
  File "/home/test/web-platform-tests/_venv2/lib/python2.7/site-packages/marionette_driver/decorators.py", line 26, in _
    return func(*args, **kwargs)
  File "/home/test/web-platform-tests/_venv2/lib/python2.7/site-packages/marionette_driver/marionette.py", line 602, in _send_message
    self._handle_error(err)
  File "/home/test/web-platform-tests/_venv2/lib/python2.7/site-packages/marionette_driver/marionette.py", line 622, in _handle_error
    raise errors.lookup(error)(message, stacktrace=stacktrace)
NoSuchWindowException: Unable to locate window: undefined
stacktrace:
    WebDriverError@chrome://marionette/content/error.js:175:5
    NoSuchWindowError@chrome://marionette/content/error.js:409:5
    GeckoDriver.prototype.switchToWindow@chrome://marionette/content/driver.js:1602:11
    despatch@chrome://marionette/content/server.js:305:40
    execute@chrome://marionette/content/server.js:275:16
    onPacket/<@chrome://marionette/content/server.js:248:20
    onPacket@chrome://marionette/content/server.js:249:9
    _onJSONObjectReady/<@chrome://marionette/content/transport.js:501:20

They've been failing since July 23rd (we didn't notice for a while since wpt.fyi was ingesting the partially complete results; I found the start point by looking where the # of results fell from ~34k to ~17k).

[stephenmcgruer]

https://github.com/web-platform-tests/wpt/pull/24680 is possibly in the regression range, it could maybe be involved?

[Hexcles]

This only seems to happen in some shards; I've not yet determined whether it's deterministic.

[jgraham]

That change wouldn't be my first guess; this code isn't actually using geckodriver. I vaugely recall seeing this error in some other context but I can't remember any details right now.

[stephenmcgruer]

FTR, I can reproduce this on Linux with

./wpt run --channel=stable --binary=~/firefox-stable/firefox --no-manifest-update --manifest MANIFEST.json --log-mach-level=debug --log-mach=- firefox dom/historical.html

(Where ~/firefox-stable is a fresh install of Firefox 79 downloaded from Mozilla's website)

Ziran also reported what I believe to be the same issue in https://github.com/web-platform-tests/wpt/issues/24991

[stephenmcgruer]

Bisecting: git bisect start 1fdbfcc68156cd8d8a74a6cfd859b43adc3ca502 d62995e62ca8653b4cf565e61c309471b544fab8

$ git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[c9989035252440d5695c28176201b3d9f14649a9] Update marionette_driver from 3.0.0 to 3.1.0

Confirmed reverting c9989035252440d5695c28176201b3d9f14649a9 on master fixes the problem.

[Hexcles]

Great job, Stephen!

Is the marionette-driver not backward compatible? Do we need to manage it the same way we manage webdriver binaries (i.e. depending on the browser version)?

[jgraham]

Let's do

handle = marionette._send_message("WebDriver:SwitchToWindow", {"handle": handle, "name": handle, "focus": True})
marionette.window = handle

marionette_driver is in maintaince mode and I think it's sometimes forgotten that wpt is an out of tree consumer. We could get this fixed on the gecko side but I think it's better in the long term to reduce our dependence on the marionette_driver package. With some luck this will all eventually be replaced by webdriver-bidi.

[Hexcles]

Can we just switch to geckodriver alone?

[jgraham]

No, that won't work. Not using geckodriver has various advantages for gecko devs (faster, they don't need to build geckodriver), plus we are using some APIs that aren't exposed in geckodriver for doing things like reftests.