search

crossbario / autobahn-testsuite

Autobahn WebSocket protocol testsuite
https://crossbar.io/autobahn/
Apache License 2.0
1k stars 84 forks source link

Python 3 support? #126

Closed ioquatix closed 3 weeks ago

ioquatix commented 1 year ago

Any chance of a plan or statement regarding python 3 support?

Even using the docker image, I see warnings, e.g.

> docker run -it --rm -v "$(pwd)/config:/config" -v "$(pwd)/reports:/reports" --name wstest crossbario/autobahn-testsuite
Unable to find image 'crossbario/autobahn-testsuite:latest' locally
latest: Pulling from crossbario/autobahn-testsuite
e22122b926a1: Pull complete 
f29e09ae8373: Pull complete 
e319e3daef68: Pull complete 
e499244fe254: Pull complete 
5a6ebed20e89: Pull complete 
8c433b46ce14: Pull complete 
bcc9ab79cfa7: Pull complete 
9e080db9f095: Pull complete 
6f8b10b35f43: Pull complete 
510a500f9039: Pull complete 
Digest: sha256:5d4ba3aa7d6ab2fdbf6606f3f4ecbe4b66f205ce1cbc176d6cdf650157e52242
Status: Downloaded newer image for crossbario/autobahn-testsuite:latest
/opt/pypy/site-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import utils, x509
...
oberstet commented 1 year ago

fwiw, I've recently added some notes here https://github.com/crossbario/autobahn-testsuite/issues/45#issuecomment-1413319820

jhoughjr commented 1 year ago

I'm wondering if these still work? I attempted to test a chrome websocket client against the docker image and nothing seems to happen. I see Entering direct command mode for peer tcp4:172.17.0.1:55784 when it opens, but no reports, no cpu usage or disk access.

When my client sent about 6k of PMCE deflate data, it seemed to throw an exception. So I made my client just connect and sit, expecting the server to do something.

The docs don't seem clear on what to expect for a successful test or what the clients are to do. I expected some kind of passing or failing reports from Chrome after 20 minutes but there is no further output. I'm on m1 if that is relevant running an amd64 docker image.

mcallegari commented 3 weeks ago

I add myself to this, since I recently integrated a Qt websocket library in my project. Autobahn is referenced from the official Qt documentation: https://doc.qt.io/qt-6/qtwebsockets-testing.html

However, I spent two hours trying to make it work on Linux without any luck. Among broken links, wrong information and Python failures, there's no way to make Autobahn work on modern Linux distribution (I tried on Ubuntu 22.04)

Can you please consider the task to port Autobahn to Python3 with a higher priority? Or at least write an up-to-date documentation to have wstest working on Python3.

Thank you!

oberstet commented 3 weeks ago

AutobahnPython works from PyPI via pip on any Python 3 (Python 2 isn't supported anymore on later releases). More so, the former is true not only for CPython but also PyPy.

AutobahnTestsuite is using internally an old snapshot of AutobahnPython - a version which still did had support for Python 2.

AutobahnTestsuite is doing a lot of crazy things internally ("fuzzing"). It has grown suprisingly complex I can tell you. Keeping the exact behavior of this crazyness exactly constant is

feature number 1!

Making the internal crazyness run on Python 3 rather than Python 2 does not add any benefit.

It introduces a big risk, I'd practically a certainty that we fail on the above feature 1 while doing the latter.

In the case, the end result would be: huge mess, confused users of AutobahnTestsuite, endless streams of "regressions" or what, broken CIs, ...

I do no think that would be welcome for anyone.

For users. And certainly not for me. I simply to not have time for such pointless (my view) exercises.

Do you @mcallegari ?

note sure if whether you saw it, but AutobahnTestsuite is recommened and used from out official Docker image. which obviously runs on any Linux.

mcallegari commented 3 weeks ago

@oberstet Thanks for you reply. So are you saying we shouldn't use AutobahnTestsuite but we should use AutobahnPython?

I'm no Python expert and the documentation often leads to "testee" which has disappeared from many places (therefore the broken links I reported above) here: https://autobahn.readthedocs.io/en/latest/websocket/examples.html#testee here: https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/websocket here: https://github.com/crossbario/autobahn-python/tree/master/examples/asyncio/websocket

I also tried to build wstest with the documentation provided here https://github.com/crossbario/autobahn-python/tree/master/wstest and it failed with numerous errors that I tried to fix but in the end, again, I got stuck on Python incompatibility. (have you considered to provide a ready-made AppImage bundle for wstest?)

So how do we get an extensive websocket conformance test report like wstest used to do on a modern Linux distribution? Sorry, but at the moment there doesn't seem to be a (working) way to test a websocket server with Autobahn.

oberstet commented 3 weeks ago

https://github.com/crossbario/autobahn-testsuite?tab=readme-ov-file#using-the-testsuite-docker-image

oberstet commented 3 weeks ago

as a user of AutobahnTestsuite, you can (and should) completely forget about anything Python.

doesn't matter.

run the Docker image above which works both for testee = websocket server and for client

done.

ioquatix commented 3 weeks ago

If anyone is interested in a working example of a github workflow: https://github.com/socketry/protocol-websocket/blob/main/.github/workflows/autobahn-server-tests.yaml

ioquatix commented 3 weeks ago

I think the main concern is that as Python 2 is EOL and eventually even this docker image may stop working.

oberstet commented 3 weeks ago

and eventually even this docker image may stop working

why?

access to the binary bits starting from gcc right up into AutobahnPython will be possible to be reproduced exactly - forever (GOAL) is sth we ideally would like to aim at. and the current status is - in principle - not far away .. I'd say .. of course this is a super complex / deep Q and I would not bet my life on many aspects (in reality as of today, vs that would be achievable in principle) ...

this is of course the crux of the matter anyways:

  1. repeatability of tests,
  2. reproducability of test results and
  3. stability of tests tools

are all essential.

if one is missing, of what value are (any) tests then anyways?

I mean, "value" for the app developer, with the app code under development being the main/only thing of interest (to the app developer)

ioquatix commented 3 weeks ago

In my experience, assuming things like docker will work the same way forever is probably unrealistic.

My main concern was this warning:

/opt/pypy/site-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography import utils, x509

While the image may work, eventually a new version of TLS may be released, or something else may change, and that will be practically impossible to support, etc. If Python 3 support is not planned, feel free to close this issue.

oberstet commented 3 weeks ago

While the image may work, eventually a new version of TLS may be released, or something else may change

sure!

but that does not change 1 bit in the image that I do have.

and the bits for that image reside on my local disk. and copies reside on N disks around the globe. and ensuring we are talking about the exact same bits of said image is easy. SHA256. done.

did I miss anything?

oberstet commented 3 weeks ago

taking your example, when we update the AutobahnTestsuite image to use a newer base OS, a newer TLS library (OpenSSL) therein, a new Python run-time - and thus not change anything at the Python library or above level, the question that matters then (rgd whether AutobahnTestsuite continues to work) is:

Do the pinned versions of Python and Python TLS and Python XYZ and AutobahnTestsuite vN.M all still work with the updated OS base?

This of course is not always true.

If it is true, then everything (the exact pinned versions of Python libraries and above) could be said to be "forward compatible" with respect to the new base OS versions (everything Python run-time and below).

of course one can think of this CUT through the tech stack at which level one targets this rolling base / pinned application approach differently within the stack:

----- CUT; what the AutobahnTestsuite project follows (e.g. w.r.t. our Docker image based release approach)

ioquatix commented 3 weeks ago

All that's totally fine..

I'm glad you are maintaining this in a sustainable way that suits you. I'm grateful for this test suite and I'll continue to use it as is for as long as possible.

However I can imagine over time, Python 2 will become a hindrance to outside contributions. For example, I did run into issues trying to make this work without docker, and I feel like that is a barrier to entry for contributing to this project. However, that may not be an issue - again totally fine.

If I had any suggestion it would simply be to clarify this, e.g. in addition to this:

Right now we only support Python 2 and Python 3 will not work

Simply state that Python 3 support is not planned.

oberstet commented 3 weeks ago

Simply state that Python 3 support is not planned.

even more precise: we only support our own official Docker images, and "support" done by "we" as in me and some guys around is on a "as we like / maybe" basis

become a hindrance to outside contributions

the biggest hindrance in my impression to outside contributions to AutobahnTestsuite is not anything related to what you hint at, but simply having the know-how (about RFC6455) and being able to understand and navigate the code base (which is pretty tricky)

porting to python 3 does not help at all with that, but would add yet one more dimension to the puzzle

in principle of course I agree! should mention that;) I am just trying to get that ideal goal aligned with reality (e.g. available developer resources, risks, ..)

ioquatix commented 3 weeks ago

I implemented WebSockets client and server for Ruby, including both HTTP/1 and HTTP/2 support. So I'm not unqualified regarding RFC6455 and navigating a reasonable code base. I don't think the code base was particularly tricky and I think the code organisation and tests are pretty good actually.

the biggest hindrance in my impression to outside contributions to AutobahnTestsuite is not anything related to what you hint at,

This is absolutely a problem for me, but it's okay if that's not a priority for you. I simply gave up trying to set up an appropriate development environment.

oberstet commented 3 weeks ago

Just to leave no wrong impression: