Chromedriver - http incompatibility with hyper

[a-nickol]

Using the current chromedriver 125.0.6422.141 with MacOS Sonoma 14.5 leads to issues in the test suite of thirtyfour:

e.g.

running 5 tests
test actions_key ... FAILED
test actions_mouse ... FAILED
test actions_drag_and_drop ... FAILED
test actions_mouse_move ... FAILED
test actions_release ... FAILED

failures:

---- actions_key stdout ----
thread 'actions_key' panicked at thirtyfour/tests/common.rs:102:51:
Failed to create WebDriver: HttpError("error sending request for url (http://localhost:9515/session)")

---- actions_mouse stdout ----
thread 'actions_mouse' panicked at thirtyfour/tests/common.rs:102:51:
Failed to create WebDriver: HttpError("error sending request for url (http://localhost:9515/session)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- actions_drag_and_drop stdout ----
thread 'actions_drag_and_drop' panicked at thirtyfour/tests/common.rs:102:51:
Failed to create WebDriver: HttpError("error sending request for url (http://localhost:9515/session)")

---- actions_mouse_move stdout ----
thread 'actions_mouse_move' panicked at thirtyfour/tests/common.rs:102:51:
Failed to create WebDriver: HttpError("error sending request for url (http://localhost:9515/session)")

---- actions_release stdout ----
thread 'actions_release' panicked at thirtyfour/tests/common.rs:102:51:
Failed to create WebDriver: HttpError("error sending request for url (http://localhost:9515/session)")

failures:
    actions_drag_and_drop
    actions_key
    actions_mouse
    actions_mouse_move
    actions_release

test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

Unfortunately I have the same issues with my applications using thiryfour. I haven't figured out the root cause of the problem yet.

Nevertheless the problem is probably out of scope of thirtyfour. But just wanted to report it here, since it is the only impact so far for me.

[a-nickol]

Tracing log from application:

2024-06-12T04:44:57.664574Z DEBUG run_webdriver_cmd: thirtyfour::session::http: webdriver request: POST /session {"capabilities":{"firstMatch":[{}],"alwaysMatch":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless"]}}},"desiredCapabilities":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless"]}}}
2024-06-12T04:44:57.665245Z TRACE run_webdriver_cmd: hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:9515)
2024-06-12T04:44:57.665317Z DEBUG run_webdriver_cmd: reqwest::connect: starting new connection: http://localhost:9515/    
2024-06-12T04:44:57.665376Z TRACE run_webdriver_cmd: hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(9515))
2024-06-12T04:44:57.665447Z DEBUG hyper_util::client::legacy::connect::dns: resolving host="localhost"
2024-06-12T04:44:57.666834Z DEBUG run_webdriver_cmd: hyper_util::client::legacy::connect::http: connecting to [::1]:9515
2024-06-12T04:44:57.666947Z TRACE run_webdriver_cmd: mio::poll: registering event source with poller: token=Token(5815450240), interests=READABLE | WRITABLE    
2024-06-12T04:44:57.667228Z DEBUG run_webdriver_cmd: hyper_util::client::legacy::connect::http: connected to [::1]:9515
2024-06-12T04:44:57.667275Z TRACE run_webdriver_cmd: hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2024-06-12T04:44:57.667379Z TRACE run_webdriver_cmd: hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:9515)
2024-06-12T04:44:57.885195Z TRACE mio::poll: deregistering event source from poller

Test that chromedriver still works:

% curl --location 'localhost:9515/session' \
--header 'Content-Type: application/json' \
--data '{"capabilities":{"firstMatch":[{}],"alwaysMatch":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless"]}}},"desiredCapabilities":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless"]}}}'

{"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome-headless-shell","browserVersion":"125.0.6422.142","chrome":{"chromedriverVersion":"125.0.6422.141 (...-refs/branch-heads/6422@{#1186})","userDataDir":"/var/folders/jd/../T/.org.chromium.Chromium.f5QcPT"},"fedcm:accounts":true,"goog:chromeOptions":{"debuggerAddress":"localhost:51113"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"mac","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"..."}}

If is use a mitmdump -p 9515 --mode reverse:http://localhost:9516 -vvv between those two applications the connection works.

This is the debug log of chromedriver: https://gist.github.com/a-nickol/07c49914a43c76f0219ecf3f6a2efa9d

% rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)
% cargo --version
cargo 1.78.0 (54d8815d0 2024-03-26)

[a-nickol]

Workaround for me so far is to route the traffic through https://mitmproxy.org/ or switch to firefox & geckodriver.

[a-nickol]

This is the root error which is thrown by hyper within the Client::decoder-method which checks:

        // According to https://tools.ietf.org/html/rfc7230#section-3.3.3
        // 1. HEAD responses, and Status 1xx, 204, and 304 cannot have a body.
        // 2. Status 2xx to a CONNECT cannot have a body.
        // 3. Transfer-Encoding: chunked has a chunked body.
        // 4. If multiple differing Content-Length headers or invalid, close connection.
        // 5. Content-Length header has a sized body.
        // 6. (irrelevant to Response)
        // 7. Read till EOF.

if inc.headers.contains_key(header::CONTENT_LENGTH) {
            debug!("illegal Content-Length header");
            Err(Parse::content_length_invalid())
}

Without the check those actions tests pass, but other tests have problems. It seems like the current chromedriver has some issues with http.

[Vrtgs]

I am unable to reproduce this issue on the current version with my M1 mac, nor in CI, make sure that the port is set because newer versions of chrome driver use a random port.

[a-nickol]

Since I have tested it with curl and had a working workaround and could make the adaptions in hyper to make it compatible proved that this has no relation with a random port. Nevertheless if this works now again seems to show that this has been fixed in chromedriver probably. I have switched to geckodriver.