aio-libs / aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
https://docs.aiohttp.org
Other
15.18k stars 2.02k forks source link

aiohttp doesn't comply autobahn test suite #4247

Open asvetlov opened 5 years ago

asvetlov commented 5 years ago

Thanks @Nothing4You for raising the question and @njsmith for the fix suggestion.

are autobahn tests not used anymore?

Richard Schwab @Nothing4You Oct 20 21:30
also, is ws_connect only supported using a ClientSession?

Andrew Svetlov @asvetlov 01:26
autobahn workflow required manual steps. If you want to integrate them into aiohttp test suite -- you are welcome!
is ws_connect only supported using a ClientSession

Sorry, I don't follow. What did you expect? ClientSession is the root for client-side API

Richard Schwab @Nothing4You 01:52
forgot that a session is also required for other stuff
thought it was just for convenience
i just noticed that the autobahn tests have code that doesn't work anymore

Nathaniel J. Smith @njsmith 06:19
running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance

At least we should run autobahn manually and fix it; ideally automate it.

webknjaz commented 3 years ago
running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance

@anesabml I think this could be a starting point for automating the test runs in our CIs :point_up:

anesabml commented 3 years ago

@webknjaz it's a great example, and it got me thinking if we could just use docker-py / python-on-whales instead of pytest-docker-compose to start the docker containers. (I am sorry if this is taking too long but I just want to make sure that I am doing the right thing)

webknjaz commented 3 years ago

Yeah, maybe. Honestly, I'd be happy to avoid a vendor lock and make sure that it also works with a rootless podman too. But if the libs don't support it, hitting up docker directly is fine by me.

Dreamsorcerer commented 2 years ago

What's still needed to get these tests passing? It's rather a shame that >10% of our CI time is spent running xfail tests. Maybe it's better to disable these tests for now? Or, only run them in specific scenarios (like push to master, so it doesn't run on PRs)?