Open asvetlov opened 5 years ago
running the autobahn tests is kind of annoying, but wsproto has an automated harness that you might be able to steal parts of: https://github.com/python-hyper/wsproto/tree/master/compliance
@anesabml I think this could be a starting point for automating the test runs in our CIs :point_up:
@webknjaz it's a great example, and it got me thinking if we could just use docker-py / python-on-whales instead of pytest-docker-compose to start the docker containers. (I am sorry if this is taking too long but I just want to make sure that I am doing the right thing)
Yeah, maybe. Honestly, I'd be happy to avoid a vendor lock and make sure that it also works with a rootless podman too. But if the libs don't support it, hitting up docker directly is fine by me.
What's still needed to get these tests passing? It's rather a shame that >10% of our CI time is spent running xfail tests. Maybe it's better to disable these tests for now? Or, only run them in specific scenarios (like push to master, so it doesn't run on PRs)?
Thanks @Nothing4You for raising the question and @njsmith for the fix suggestion.
At least we should run autobahn manually and fix it; ideally automate it.