autobahn test failures - Githubissues

python-trio / trio-websocket

WebSocket client and server implementation for Python Trio

MIT License

70 stars 25 forks source link

autobahn test failures #147

Closed belm0 closed 3 years ago

belm0 commented 3 years ago

Dozens of test cases are failing in the client suite.

All the cases I've checked are due to "The connection was failed by the wrong endpoint (FAILED)".

I haven't been able to run the server suite.

belm0 commented 3 years ago

Regarding running the server suite, the instructions are outdated, and don't take platform differences into consideration.

in the docker invocation, I needed to open the port to the server running on the host with -p 9000:9000
on OS X or Windows, the host in fuzzingclient.json must be changed from 172.17.0.1 to host.docker.internal

The server suite also has dozens of failures with "The connection was failed by the wrong endpoint (FAILED)".

belm0 commented 3 years ago

I went back to trio-websocket 0.6.0 release, and client test suite still fails on a number of tests with "The connection was failed by the wrong endpoint (FAILED)":

9.1.{4,5,6}
9.2.{4,5,6}
9.{3,4}

--> these were only due to insufficient limit on message length

So for the other failures seen in head, it seems there is a regression since 0.6.0.

belm0 commented 3 years ago

All I know so far is that there's something nondeterministic about the client shutdown on error. Sometimes it matches the test case expectation on the server end, sometimes not.

belm0 commented 3 years ago

The regression seems to be from trio-websocket 0.7.0-- specially PR #106, which was a significant refactoring for upgrade to wsproto 0.14 API.