Headless tests failing on Travis CI

[jmmv]

🐛 Bug description

I have a project where I use Travis CI for testing. As of a few weeks ago, all of my wasm-pack test invocations started failing and I haven't been able to reproduce the problem locally nor I have been able to diagnose what's going on yet.

The project in question is at https://github.com/jmmv/endbasic/tree/fixes, and I'm linking to the branch I'm currently using to try to diagnose what's going on in Travis CI. Travis CI is configured to use Ubuntu Xenial, downloading the latest versions of Rust and wasm-pack.

I am using wasm-pack test --headless --chrome. I do not install the Chome webdriver beforehand so that wasm-pack can download whatever it pleases. (But I also tried to install it from Xenial's repositories, and no luck.)

Whenever I try my test invocations on a macOS or a Debian system, everything works fine. I cannot tell what exactly is different that causes the builds to fail remotely.

I'm suspecting that #886 is related and I've been trying to workaround it by enabling mutable goals, but then hit other problems with the --headless --firefox variant not working due to an "Invalid Content-Type". So I switched to Chrome and I'm still seeing problems.

🤔 Expected Behavior

This build was working fine on its own until my CI picked up a new version of something. I'm suspecting wasm-bindgen post-0.2.66, but it'd also be wasm-pack. I would also have expected this to resolve by itself after other people reported this weeks ago, but it hasn't.

👟 Steps to reproduce

Take a look at https://github.com/jmmv/endbasic/tree/fixes and the corresponding results for that branch in Travis CI. I've tried to instrument the build to add RUST_LOG=wasm_bindgen_test_runner and got the following information:

[INFO]: Installing wasm-bindgen...

[INFO]: Getting chromedriver...

   Compiling endbasic-web v0.2.0 (/home/travis/build/jmmv/endbasic/web)

    Finished test [unoptimized + debuginfo] target(s) in 1.01s

     Running /home/travis/build/jmmv/endbasic/target/wasm32-unknown-unknown/debug/deps/endbasic_web-f99e29e45d414a8d.wasm

Set timeout to 20 seconds...

[2020-08-29T03:19:13Z DEBUG wasm_bindgen_test_runner::headless] executing "/home/travis/.cache/.wasm-pack/chromedriver-6b6c392e84aa6814/chromedriver" "--port=36613"

Running headless tests in Chrome on `http://127.0.0.1:36613/`

Try find `webdriver.json` for configure browser's capabilities:

Not found

[2020-08-29T03:19:13Z DEBUG wasm_bindgen_test_runner::headless] POST /session {"desiredCapabilities":{"goog:chromeOptions":{"args":["headless","disable-dev-shm-usage","no-sandbox"]}},"requiredCapabilities":{}}

[2020-08-29T03:19:14Z DEBUG wasm_bindgen_test_runner::headless] got: {"sessionId":"bcc4a4fd58fda8daa3e871f23f55d5e4","status":33,"value":{"message":"session not created: This version of ChromeDriver only supports Chrome version 85\n  (Driver info: chromedriver=85.0.4183.87 (cd6713ebf92fa1cacc0f1a598df280093af0c5d7-refs/branch-heads/4183@{#1689}),platform=Linux 4.15.0-1028-gcp x86_64)"}}

[2020-08-29T03:19:14Z DEBUG wasm_bindgen_test_runner::headless] POST /session/bcc4a4fd58fda8daa3e871f23f55d5e4/url {"url":"http://127.0.0.1:40765"}

[2020-08-29T03:19:14Z DEBUG wasm_bindgen_test_runner::headless] DELETE /session/bcc4a4fd58fda8daa3e871f23f55d5e4/window

[2020-08-29T03:19:14Z WARN  wasm_bindgen_test_runner::headless] failed to close window non-200 response code: 404

    {"value":{"error":"invalid session id","message":"invalid session id","stacktrace":"#0 0x556376030a39 \u003Cunknown>\n"}}

driver status: signal: 9

driver stdout:

    Starting ChromeDriver 85.0.4183.87 (cd6713ebf92fa1cacc0f1a598df280093af0c5d7-refs/branch-heads/4183@{#1689}) on port 36613

    Only local connections are allowed.

    Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.

    ChromeDriver was started successfully.

driver stderr:

    [1598671153.756][SEVERE]: bind() failed: Cannot assign requested address (99)

Error: non-200 response code: 404

{"value":{"error":"invalid session id","message":"invalid session id","stacktrace":"#0 0x556376030a39 \u003Cunknown>\n"}}

error: test failed, to rerun pass '--lib'

Error: Running Wasm tests with wasm-bindgen-test failed

Caused by: failed to execute `cargo test`: exited with exit code: 1

  full command: "cargo" "test" "--target" "wasm32-unknown-unknown"

The command "./admin/travis-build.sh" exited with 1.

This isn't helping me a lot. The error says that this ChromeDriver only supports version 85... but the reported version is 85?

🌍 Your environment

wasm-pack version: 0.9.1 rustc version: 1.46.0

[jmmv]

Aha, I finally figured this out after installing a Ubuntu Xenial VM (which is ancient at this point, something I hadn't realized) and playing with it directly.

The problem arises either because Chrome itself (not the driver) isn't installed or because the installed version is too old. Running apt update && apt install chromium-browser chromium-chromedriver fixed the problems for me, as then I get version 84 of each component and they are in sync.

But... that didn't fix the problem on Travis even though I ran the same setup commands. Something else must be getting in the way but I don't know what it is. Replacing manual apt commands with a request to install the latest Chrome version via addons fixes the issue there.

So... "my fault" I guess, but I think the error messages printed by the tool would do a better job at pointing at the problem. Printing both the version of the driver and the browser, as well as which specific binaries are in use (paths), would help a lot here.

[WorldSEnder]

I've been bitten by this also, on my local machine. After update chrome to the latest version, everything was smooth sailing. I would love a sanity check for version compatibility of the detected binaries, resulting in a better error message.