webdriver/tests/bidi failing in Edge

[bwalderman]

None of the WebDriver BiDi tests are passing in Edge, even though they are passing in Chrome: https://wpt.fyi/results/webdriver/tests/bidi?label=experimental&label=master&aligned

The Edge tests run on a Windows-based Azure pipeline defined in https://github.com/web-platform-tests/wpt/blob/master/.azure-pipelines.yml

When running the tests locally on Windows, it looks like they pass locally on Python 3.8, but fail on Python 3.10 which is the current version used in WPT:

    async def close(self, code: int = 1000, reason: str = "") -> None:
        """
        Perform the closing handshake.

        :meth:`close` waits for the other end to complete the handshake and
        for the TCP connection to terminate. As a consequence, there's no need
        to await :meth:`wait_closed`; :meth:`close` already does it.

        :meth:`close` is idempotent: it doesn't do anything once the
        connection is closed.

        Wrapping :func:`close` in :func:`~asyncio.create_task` is safe, given
        that errors during connection termination aren't particularly useful.

        Canceling :meth:`close` is discouraged. If it takes too long, you can
        set a shorter ``close_timeout``. If you don't want to wait, let the
        Python process exit, then the OS will close the TCP connection.

        :param code: WebSocket close code
        :param reason: WebSocket close reason

        """
        try:
>           await asyncio.wait_for(
                self.write_close_frame(serialize_close(code, reason)),
                self.close_timeout,
                loop=self.loop if sys.version_info[:2] < (3, 8) else None,
            )
E           TypeError: wait_for() got an unexpected keyword argument 'loop'

code       = 1000
reason     = ''
self       = <websockets.client.WebSocketClientProtocol object at 0x00000205C9EBE200>

tools\third_party\websockets\src\websockets\protocol.py:680: TypeError

It's not clear why this is a problem on Windows, but not on Linux where the Chrome tests are run. This issue might eventually be addressed by https://github.com/web-platform-tests/wpt/pull/36524. When I check out out that branch locally though, I run into a different error that seems related to how the test fixture is passing the bidi_session object to the test function:

FAIL test_websocket_url_connect[capabilities0] - AttributeError: 'async_generator' object has no attribute 'capabilities'
session = <async_generator object session at 0x00000195043C3840>

    @pytest.mark.asyncio
    @pytest.mark.capabilities({"webSocketUrl": True})
    async def test_websocket_url_connect(session):
>       websocket_url = session.capabilities["webSocketUrl"]
E       AttributeError: 'async_generator' object has no attribute 'capabilities'

session    = <async_generator object session at 0x00000195043C3840>

[bwalderman]

CC @mathiasbynens @sadym-chromium @foolip @jgraham

[sadym-chromium]

@bwalderman please make sure you use the latest WPT revision, as the mentioned error should be fixed in https://github.com/web-platform-tests/wpt/pull/36372 My bad. It's the same issue, but not fixed yet.

[sadym-chromium]

I created PR which should fix the issue: https://github.com/web-platform-tests/wpt/pull/36787

@bwalderman do you have a possibility to try it out on Azure?