CI: vmprof on Python 3.11

[om26er]

Python 3.10 has been out for a few days. Crossbar currently does not support it. There are multiple issues

1. vmprof doesn't install from pypi. However install from master works (reported https://github.com/vmprof/vmprof-python/issues/240)
2. If I install crossbar's deps without vmprof, it still does't start because of below issues
   • from collections import Mapping needs to be replaced with from collections.abc import Mapping (see https://github.com/crossbario/crossbar/blob/712d9da9173829be7df3ecba5dbb7981dd648fcd/crossbar/edge/personality.py#L10 (will do a PR fix that))

After fixing the above imports, the next issue comes with cython

(venv) om26er@HomePC:~/scm/crossbario$ crossbar start

  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/autobahn/xbr/__init__.py", line 46, in <module>
    import eth_account
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/eth_account/__init__.py", line 1, in <module>
    from eth_account.account import (  # noqa: F401
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/eth_account/account.py", line 8, in <module>
    from cytoolz import (
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cytoolz/__init__.py", line 3, in <module>
    from .functoolz import *
/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cytoolz/functoolz.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_SendTraceback (most recent call last):
  File "/home/om26er/scm/foobar/markhor/venv/bin/crossbar", line 33, in <module>
    sys.exit(load_entry_point('crossbar', 'console_scripts', 'crossbar')())
  File "/home/om26er/scm/crossbario/crossbar/crossbar/__init__.py", line 280, in run
    from crossbar import edge
  File "/home/om26er/scm/crossbario/crossbar/crossbar/edge/__init__.py", line 13, in <module>
    from crossbar.edge.personality import Personality
  File "/home/om26er/scm/crossbario/crossbar/crossbar/edge/personality.py", line 24, in <module>
    from crossbar.edge.worker.realmstore import CfxDbRealmStore
  File "/home/om26er/scm/crossbario/crossbar/crossbar/edge/worker/realmstore.py", line 21, in <module>
    import cfxdb
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cfxdb/__init__.py", line 13, in <module>
    from .common import address, uint256, unpack_uint256, pack_uint256,\
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cfxdb/common.py", line 8, in <module>
    import web3
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/web3/__init__.py", line 6, in <module>
    from eth_account import (
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/eth_account/__init__.py", line 1, in <module>
    from eth_account.account import (  # noqa: F401
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/eth_account/account.py", line 8, in <module>
    from cytoolz import (
  File "/home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cytoolz/__init__.py", line 3, in <module>
    from .functoolz import *
ImportError: /home/om26er/scm/foobar/markhor/venv/lib/python3.10/site-packages/cytoolz/functoolz.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

[oberstet]

interesting. thanks for digging into the individual issues! on a side note, I'm wondering why 3.10 point release triggers such a bunch of issues. but then, anyways;)

from collections import Mapping, Sequence

ok, this is probably the one most surprising to me of the 3. what's the problem with that?

cytoolz/functoolz.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

again, Python ABI :( cytoolz is like cffi and such stuff highly dependent on all ABI .. of CPython. which reminds me: what will PyPy do with all these issues?

[om26er]

ok, this is probably the one most surprising to me of the 3. what's the problem with that?

om26er@RemotePC:~$ python3.10 
Python 3.10.0rc2 (default, Sep 10 2021, 14:59:37) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from collections import Mapping
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
>>>
om26er@RemotePC:~$ python3
Python 3.9.7 (default, Sep 10 2021, 14:59:43) 
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from collections import Mapping
<stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
>>>

[tirkarthi]

I have created https://github.com/crossbario/crossbar/pull/1897 to fix the collections issue and another Python 3.11 issue. Importing ABC from collections was deprecated from Python 3.4 and was removed in Python 3.10 to use collections.abc module instead.

[oberstet]

thanks for contributing! since you're first time contributor to this project, the PR didn't start the CI automatically - done that now!

rgd vmprof and python 3.10 ^: @fijal any opinions? I mean, rather than making vmprof optional (again) in crossbar, actually making it work would be much preferred;)

[fijal]

vmprof is a bit unmaintained I'm afraid :/ I can probably look next week, maybe we can do a new release

[oberstet]

^ thanks for the note! "bit unmaintained": it's unfortunate:( open source funding - same here. well.

[om26er]

This new tool was recently open-sourced https://github.com/bloomberg/memray, I think that is something to investigate as well.

Here is a brief summary by Yury https://twitter.com/1st1/status/1516859294896906241

[oberstet]

This new tool was recently open-sourced https://github.com/bloomberg/memray, I think that is something to investigate as well.

Interesting! Does it work with PyPy?

[oberstet]

this works on cpython 3.11:

pip install "vmprof @ git+https://github.com/matthiasdiener/vmprof-python.git@py311#egg=vmprof"

see:

• https://github.com/vmprof/vmprof-python/pull/251
• https://github.com/vmprof/vmprof-python/issues/240