search

crossbario / autobahn-python

WebSocket and WAMP in Python for Twisted and asyncio
https://crossbar.io/autobahn
MIT License
2.48k stars 763 forks source link

Msgpack serializer not available on PyPy based container #1511

Open om26er opened 2 years ago

om26er commented 2 years ago

When running PyPy, the msgpack serializer expects umsgpack to be installed. It seems that is missing in the docker container.

om26er@RemotePC:~$ docker run -it crossbario/autobahn-python:pypy-amd64-21.11.1-20211223-3a8caa6 python3 -c "from autobahn.wamp.serializer import MsgPackSerializer"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'MsgPackSerializer' from 'autobahn.wamp.serializer' (/opt/pypy/lib/pypy3.8/site-packages/autobahn/wamp/serializer.py)
om26er@RemotePC:~$ docker run -it crossbario/autobahn-python:pypy-amd64-21.11.1-20211223-3a8caa6 python3 -c "import umsgpack"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'umsgpack'
oberstet commented 2 years ago

so the dockerfile reference the right AB dist

https://github.com/crossbario/autobahn-python/blob/3a8caa65170d35c6d3de6ea8154d8c7d5c1b6e03/docker/Dockerfile.pypy-slim#L77

but the deps added for that dist for pypy aren't enough

https://github.com/crossbario/autobahn-python/blob/3a8caa65170d35c6d3de6ea8154d8c7d5c1b6e03/setup.py#L74

this is also reflected in

(cpy310_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python/docker$ docker run -it crossbario/autobahn-python:pypy-amd64-21.11.1-20211223-3a8caa6 
................................................................................
Python version:

    Host machine           : x86_64
    Operating system       : Linux-5.4.0-91-generic-x86_64-with-glibc2.2.5
    Python                 : 3.8.12/PyPy-7.3.7
    Executable             : /opt/pypy/bin/pypy3
................................................................................
Autobahn version:

    autobahn               : v21.11.1
    autobahn.xbr.HAS_XBR   : True
................................................................................
Autobahn build environment:

    AUTOBAHN_BUILD_DATE    : 2021-12-23
    AUTOBAHN_VERSION       : 21.11.1
    AUTOBAHN_VCS_REF       : 3a8caa6
    AUTOBAHN_BUILD_ID      : 20211223-3a8caa6
................................................................................
Autobahn dependencies and (optional) components:

    txaio                  : 21.2.1

    UTF8 Validator         : autobahn
    XOR Masker             : autobahn
    JSON Codec             : stdlib
    MsgPack Codec          : None
    CBOR Codec             : cbor-1.0.0
    UBJSON Codec           : ubjson-0.16.1
    FlatBuffers            : flatbuffers-2.0

    Twisted                : 21.7.0-EPollReactor
    LMDB                   : 1.2.1/lmdb-0.9.29
    zLMDB                  : 21.2.1
    NumPy                  : 1.21.5
................................................................................

 XBR CLI v21.11.1

   Contract addresses:

      XBRToken   : 0xaCef957D54c639575f4DB68b1992B36504f33FEA [source: builtin]
      XBRNetwork : 0x7A3d22c59e8F8f1b88ba7205f3f5a65Bc86D04Bc [source: builtin]
      XBRDomain  : 0xf5fb56886f033855C1a36F651E927551749361bC [source: builtin]
      XBRCatalog : 0x2C77E46Ea9502B363343e8c826c41c7fdb25Db66 [source: builtin]
      XBRMarket  : 0x0DcF924ab0846101d31514E9fb3adf5070d4B83d [source: builtin]
      XBRChannel : 0x670497A012322B99a5C18B8463940996141Cb952 [source: builtin]

(cpy310_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python/docker$
oberstet commented 2 years ago

mmh, it does work with dist==all though

(base) oberstet@intel-nuci7:~$ wget https://downloads.python.org/pypy/pypy3.8-v7.3.7-linux64.tar.bz2
--2021-12-29 20:46:09--  https://downloads.python.org/pypy/pypy3.8-v7.3.7-linux64.tar.bz2
Auflösen des Hostnamens downloads.python.org (downloads.python.org) … 151.101.192.175, 151.101.128.175, 151.101.64.175, ...
Verbindungsaufbau zu downloads.python.org (downloads.python.org)|151.101.192.175|:443... verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 31013325 (30M) [application/x-tar]
Wird in »pypy3.8-v7.3.7-linux64.tar.bz2« gespeichert.

pypy3.8-v7.3.7-linux64.tar.bz2    100%[=============================================================>]  29,58M  11,5MB/s    in 2,6s    

2021-12-29 20:46:11 (11,5 MB/s) - »pypy3.8-v7.3.7-linux64.tar.bz2« gespeichert [31013325/31013325]

(base) oberstet@intel-nuci7:~$ tar xvf pypy3.8-v7.3.7-linux64.tar.bz2 
pypy3.8-v7.3.7-linux64/
...
(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ pip install -e .[all]
...
  Running setup.py develop for autobahn
Successfully installed Automat-20.2.0 MarkupSafe-2.0.1 PyGObject-3.42.0 aiohttp-3.8.1 aiosignal-1.2.0 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 async-timeout-4.0.2 attrs-21.4.0 autobahn base58-2.1.1 bitarray-1.2.2 cached-property-1.5.2 cbor-1.0.0 cbor2-5.4.2 certifi-2021.10.8 charset-normalizer-2.0.9 click-8.0.3 constantly-15.1.0 cryptography-36.0.1 cytoolz-0.11.2 ecdsa-0.17.0 eth-abi-2.1.1 eth-account-0.5.6 eth-hash-0.3.2 eth-keyfile-0.5.1 eth-keys-0.3.3 eth-rlp-0.2.1 eth-typing-2.2.2 eth-utils-1.10.0 flatbuffers-2.0 frozenlist-1.2.0 hexbytes-0.2.2 hkdf-0.0.3 hyperlink-21.0.0 idna-3.3 incremental-21.3.0 ipfshttpclient-0.8.0a2 jinja2-3.0.3 jsonschema-3.2.0 lmdb-1.2.1 lru-dict-1.1.7 mnemonic-0.20 morphys-1.0 multiaddr-0.0.9 multidict-5.2.0 mypy-extensions-0.4.3 netaddr-0.8.0 numpy-1.21.5 parsimonious-0.8.1 passlib-1.7.4 protobuf-3.19.1 py-ecc-5.2.0 py-eth-sig-utils-0.4.0 py-multihash-2.0.1 py-ubjson-0.16.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycairo-1.20.1 pycryptodome-3.12.0 pynacl-1.4.0 pyopenssl-21.0.0 pyqrcode-1.2.1 pyrsistent-0.18.0 python-snappy-0.6.0 pytrie-0.4.0 pyyaml-6.0 requests-2.26.0 rlp-2.0.1 service-identity-21.1.0 six-1.16.0 sortedcontainers-2.4.0 spake2-0.8 toolz-0.11.2 twisted-21.7.0 txaio-21.2.1 typing-extensions-4.0.1 u-msgpack-python-2.7.1 urllib3-1.26.7 varint-1.0.2 web3-5.25.0 websockets-9.1 xbr-21.2.1 yapf-0.29.0 yarl-1.7.2 zlmdb-21.2.1 zope.interface-5.4.0
WARNING: You are using pip version 21.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/home/oberstet/pypy38_1/bin/python -m pip install --upgrade pip' command.
(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -c "from autobahn.wamp.serializer import MsgPackSerializer"
(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -c "import umsgpack"
(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -V
Python 3.8.12 (9ef55f6fc369, Oct 24 2021, 20:11:54)
[PyPy 7.3.7 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ which python
/home/oberstet/pypy38_1/bin/python
oberstet commented 2 years ago

mmh, this is strange. it also works with only dist==serialization:

(pypy38_1) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ source ~/pypy38_2/bin/activate
(pypy38_2) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ pip install -e .[serialization]
Obtaining file:///home/oberstet/scm/crossbario/autobahn-python
Requirement already satisfied: txaio>=21.2.1 in /home/oberstet/pypy38_2/lib/pypy3.8/site-packages (from autobahn==21.11.1) (21.2.1)
Requirement already satisfied: cryptography>=3.4.6 in /home/oberstet/pypy38_2/lib/pypy3.8/site-packages (from autobahn==21.11.1) (36.0.1)
Requirement already satisfied: hyperlink>=21.0.0 in /home/oberstet/pypy38_2/lib/pypy3.8/site-packages (from autobahn==21.11.1) (21.0.0)
Requirement already satisfied: setuptools in /home/oberstet/pypy38_2/lib/pypy3.8/site-packages (from autobahn==21.11.1) (56.0.0)
Collecting u-msgpack-python>=2.1
  Using cached u_msgpack_python-2.7.1-py2.py3-none-any.whl (10.0 kB)
Collecting cbor2>=5.2.0
  Using cached cbor2-5.4.2-py3-none-any.whl
Collecting cbor>=1.0.0
  Using cached cbor-1.0.0.tar.gz (20 kB)
Collecting py-ubjson>=0.16.1
  Using cached py-ubjson-0.16.1.tar.gz (50 kB)
Collecting flatbuffers>=1.12
  Using cached flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: cffi>=1.12 in /home/oberstet/pypy3.8-v7.3.7-linux64/lib/pypy3.8 (from cryptography>=3.4.6->autobahn==21.11.1) (1.14.6)
Requirement already satisfied: idna>=2.5 in /home/oberstet/pypy38_2/lib/pypy3.8/site-packages (from hyperlink>=21.0.0->autobahn==21.11.1) (3.3)
Using legacy 'setup.py install' for cbor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for py-ubjson, since package 'wheel' is not installed.
Installing collected packages: u-msgpack-python, py-ubjson, flatbuffers, cbor2, cbor, autobahn
    Running setup.py install for py-ubjson ... done
    Running setup.py install for cbor ... done
  Attempting uninstall: autobahn
    Found existing installation: autobahn 21.11.1
    Uninstalling autobahn-21.11.1:
      Successfully uninstalled autobahn-21.11.1
  Running setup.py develop for autobahn
Successfully installed autobahn cbor-1.0.0 cbor2-5.4.2 flatbuffers-2.0 py-ubjson-0.16.1 u-msgpack-python-2.7.1
WARNING: You are using pip version 21.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/home/oberstet/pypy38_2/bin/python -m pip install --upgrade pip' command.
(pypy38_2) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -c "import umsgpack"
(pypy38_2) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -c "from autobahn.wamp.serializer import MsgPackSerializer"
(pypy38_2) (base) oberstet@intel-nuci7:~/scm/crossbario/autobahn-python$ python -c "import umsgpack"
oberstet commented 2 years ago

not sure why, but sth is going wrong during the image build https://gist.github.com/oberstet/fab1c4872861827461397a6cc7c922b8 ...

oberstet commented 2 years ago

it's still there:(

docker run -it crossbario/autobahn-python:pypy-amd64-22.1.1-20220128-dc626948 \
    python3 -c "from autobahn.wamp.serializer import MsgPackSerializer"
oberstet commented 2 years ago

still there

(base) oberstet@intel-nuci7:~$ docker run -it crossbario/autobahn-python:pypy-amd64-22.3.1 \
>     python3 -c "from autobahn.wamp.serializer import MsgPackSerializer"
Unable to find image 'crossbario/autobahn-python:pypy-amd64-22.3.1' locally
pypy-amd64-22.3.1: Pulling from crossbario/autobahn-python
eff15d958d66: Pull complete 
c36ae8d36378: Pull complete 
1358f84c3b69: Pull complete 
0517128f0165: Pull complete 
1722acdad438: Pull complete 
cd964ccd0977: Pull complete 
25667c04e168: Pull complete 
17e6106f99d4: Pull complete 
c26db71fac11: Pull complete 
933c96519318: Pull complete 
Digest: sha256:a8a8f01a6e748c8eeb7c0e819bf42548a9b66ae91547f666d88cf70aa79a15ee
Status: Downloaded newer image for crossbario/autobahn-python:pypy-amd64-22.3.1
/opt/pypy/lib/pypy3.8/site-packages/numpy/core/_add_newdocs.py:2826: UserWarning: add_newdoc was used on a pure-python object <bound method __class_getitem__ of <class 'numpy.ndarray'>>. Prefer to attach it directly to the source.
  add_newdoc('numpy.core.multiarray', 'ndarray', ('__class_getitem__',
/opt/pypy/lib/pypy3.8/site-packages/numpy/core/_add_newdocs.py:6128: UserWarning: add_newdoc was used on a pure-python object <bound method __class_getitem__ of <class 'numpy.dtype'>>. Prefer to attach it directly to the source.
  add_newdoc('numpy.core.multiarray', 'dtype', ('__class_getitem__',
/opt/pypy/lib/pypy3.8/site-packages/numpy/core/_add_newdocs.py:6640: UserWarning: add_newdoc was used on a pure-python object <bound method __class_getitem__ of <class 'numpy.number'>>. Prefer to attach it directly to the source.
  add_newdoc('numpy.core.numerictypes', 'number', ('__class_getitem__',
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'MsgPackSerializer' from 'autobahn.wamp.serializer' (/opt/pypy/lib/pypy3.8/site-packages/autobahn/wamp/serializer.py)