search

crossbario / crossbar

Crossbar.io - WAMP application router
https://crossbar.io/
Other
2.05k stars 274 forks source link

prepare for v21.11.1 #1917

Closed oberstet closed 2 years ago

oberstet commented 2 years ago

couple of smaller issues .. the docs in CI run into

sphinx.errors.SphinxWarning: /home/cbbuilder/actions-runner-crossbar/_work/crossbar/crossbar/docs/Table-of-Contents.rst:8:toctree contains reference to document 'soss_licenses_table' that doesn't have a title: no link will be generated

Warning, treated as error:
/home/cbbuilder/actions-runner-crossbar/_work/crossbar/crossbar/docs/Table-of-Contents.rst:8:toctree contains reference to document 'soss_licenses_table' that doesn't have a title: no link will be generated
ERROR: InvocationError for command /opt/hostedtoolcache/Python/3.9.2/x64/bin/sphinx-build -nWT -b dummy ./docs ./docs/_build (exited with code 2)
___________________________________ summary ________________

the soss_licenses_table file is generated in make freeze, which creates requirements file and this license overview for all deps

while doing the make freeze to prepare this PR, the following appeared on my system

ERROR: eth-utils 1.10.0 has requirement eth-typing<3.0.0,>=2.2.1, but you'll have eth-typing 3.0.0 which is incompatible.
ERROR: eth-abi 2.1.1 has requirement eth-typing<3.0.0,>=2.0.0, but you'll have eth-typing 3.0.0 which is incompatible.
ERROR: eth-keys 0.3.3 has requirement eth-typing<3.0.0,>=2.2.1, but you'll have eth-typing 3.0.0 which is incompatible.
ERROR: web3 5.25.0 has requirement eth-typing<3.0.0,>=2.0.0, but you'll have eth-typing 3.0.0 which is incompatible.
ERROR: py-ecc 5.2.0 has requirement eth-typing<3.0.0,>=2.1.0, but you'll have eth-typing 3.0.0 which is incompatible.

(this could be bogus ... I didn't test in a completely fresh venv ... other stuff might interfere)

and

cat requirements-pinned.txt | xargs vers/bin/hashin > requirements.txt
No data found for version 0.4.0
make: *** [Makefile:89: freeze] Fehler 123
oberstet commented 2 years ago

in a completely newly built cpy3.10 and fresh venv:

  Building wheel for vmprof (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/oberstet/scm/crossbario/crossbar/vers/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1rh9jwvm/vmprof/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1rh9jwvm/vmprof/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-wvqhn72e --python-tag cp310
       cwd: /tmp/pip-install-1rh9jwvm/vmprof/
  Complete output (216 lines):
  running bdist_wheel
  running build
  running build_py
  checking build system type... x86_64-pc-linux-gnu
  checking host system type... x86_64-pc-linux-gnu
  checking target system type... x86_64-pc-linux-gnu
  checking for gcc... gcc
...
  checking whether strnlen is declared... yes
  checking for getexecname... no
  checking whether tests can run... yes
  configure: creating ./config.status
  config.status: creating Makefile
  config.status: creating backtrace-supported.h
  config.status: creating config.h
  config.status: config.h is unchanged
  config.status: executing libtool commands
  sed: can't read ./ltmain.sh: No such file or directory
  sed: can't read ./ltmain.sh: No such file or directory
  mv: cannot stat 'libtoolT': No such file or directory
  cp: cannot stat 'libtoolT': No such file or directory
  chmod: cannot access 'libtool': No such file or directory
  config.status: executing gstdint.h commands
  config.status: executing default commands
  creating build
  creating build/lib.linux-x86_64-3.10
  creating build/lib.linux-x86_64-3.10/vmprof
  copying vmprof/stats.py -> build/lib.linux-x86_64-3.10/vmprof
  copying vmprof/__init__.py -> build/lib.linux-x86_64-3.10/vmprof
  copying vmprof/show.py -> build/lib.linux-x86_64-3.10/vmprof
...
  copying jitlog/parser.py -> build/lib.linux-x86_64-3.10/jitlog
  running build_ext
  building '_vmprof' extension
  creating build/temp.linux-x86_64-3.10
  creating build/temp.linux-x86_64-3.10/src
  creating build/temp.linux-x86_64-3.10/src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/_vmprof.c -o build/temp.linux-x86_64-3.10/src/_vmprof.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  src/_vmprof.c: In function ‘emit_code_object’:
  src/_vmprof.c:68:13: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     68 |     co_name = PyUnicode_AsUTF8(co->co_name);
        |             ^
  src/_vmprof.c:71:17: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     71 |     co_filename = PyUnicode_AsUTF8(co->co_filename);
        |                 ^
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/compat.c -o build/temp.linux-x86_64-3.10/src/compat.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/backtrace.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/backtrace.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/dwarf.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/dwarf.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/elf.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/elf.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/fileline.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/fileline.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/mmap.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/mmap.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/mmapio.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/mmapio.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/posix.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/posix.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/sort.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/sort.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/libbacktrace/state.c -o build/temp.linux-x86_64-3.10/src/libbacktrace/state.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/machine.c -o build/temp.linux-x86_64-3.10/src/machine.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/symboltable.c -o build/temp.linux-x86_64-3.10/src/symboltable.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/oberstet/scm/crossbario/crossbar/vers/include -I/home/oberstet/cpy310/include/python3.10 -c src/vmp_stack.c -o build/temp.linux-x86_64-3.10/src/vmp_stack.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 -I src/ -I src/libbacktrace
  In file included from /home/oberstet/cpy310/include/python3.10/bytesobject.h:62,
                   from /home/oberstet/cpy310/include/python3.10/Python.h:95,
                   from src/vmprof.h:6,
                   from src/vmp_stack.h:3,
                   from src/vmp_stack.c:1:
  src/vmp_stack.c: In function ‘_write_python_stack_entry’:
  src/vmp_stack.c:102:47: error: ‘PyCodeObject’ {aka ‘struct PyCodeObject’} has no member named ‘co_lnotab’
    102 |         lnotab = PyStr_AS_STRING(frame->f_code->co_lnotab);
        |                                               ^~
  /home/oberstet/cpy310/include/python3.10/cpython/bytesobject.h:33:53: note: in definition of macro ‘PyBytes_AS_STRING’
     33 |                                 (((PyBytesObject *)(op))->ob_sval))
        |                                                     ^~
  src/vmp_stack.c:102:18: note: in expansion of macro ‘PyStr_AS_STRING’
    102 |         lnotab = PyStr_AS_STRING(frame->f_code->co_lnotab);
        |                  ^~~~~~~~~~~~~~~
  In file included from /home/oberstet/cpy310/include/python3.10/Python.h:87,
                   from src/vmprof.h:6,
                   from src/vmp_stack.h:3,
                   from src/vmp_stack.c:1:
  src/vmp_stack.c:108:52: error: ‘PyCodeObject’ {aka ‘struct PyCodeObject’} has no member named ‘co_lnotab’
    108 |             len = (int)PyStr_GET_SIZE(frame->f_code->co_lnotab);
        |                                                    ^~
  /home/oberstet/cpy310/include/python3.10/object.h:121:47: note: in definition of macro ‘_PyVarObject_CAST’
    121 | #define _PyVarObject_CAST(op) ((PyVarObject*)(op))
        |                                               ^~
  /home/oberstet/cpy310/include/python3.10/cpython/bytesobject.h:34:58: note: in expansion of macro ‘Py_SIZE’
     34 | #define PyBytes_GET_SIZE(op)  (assert(PyBytes_Check(op)),Py_SIZE(op))
        |                                                          ^~~~~~~
  src/compat.h:8:30: note: in expansion of macro ‘PyBytes_GET_SIZE’
      8 |       #define PyStr_GET_SIZE PyBytes_GET_SIZE
        |                              ^~~~~~~~~~~~~~~~
  src/vmp_stack.c:108:24: note: in expansion of macro ‘PyStr_GET_SIZE’
    108 |             len = (int)PyStr_GET_SIZE(frame->f_code->co_lnotab);
        |                        ^~~~~~~~~~~~~~
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for vmprof
  Running setup.py clean for vmprof
om26er commented 2 years ago

That's because Python 3.10 is currently not supported for Crossbar as mentioned in https://github.com/crossbario/crossbar/issues/1896

om26er commented 2 years ago

When using Python 3.10, vmprof has to be installed from master

oberstet commented 2 years ago

When using Python 3.10, vmprof has to be installed from master

ahh, right, rgd py3.10, thanks for reminding;) so what should we do for a CB release?

om26er commented 2 years ago

The mapping related issue was fixed via https://github.com/crossbario/crossbar/commit/93f3931eb3fb34bae59c6a8d3d4bcbbef7131d67

The two main issues are: vmprof and cython (see traceback on the issue)

vmprof issue has a workaround (install from master). For cython, need to investigate still

oberstet commented 2 years ago

rgd cython, just tried:

(cpy310_1) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ pip install cython
Collecting cython
  Using cached Cython-0.29.24-py2.py3-none-any.whl (979 kB)
Installing collected packages: cython
Successfully installed cython-0.29.24
(cpy310_1) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ python
Python 3.10.0 (default, Nov 24 2021, 12:27:36) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cytoolz.functoolz import *
>>> import eth_account
>>>
oberstet commented 2 years ago

so it seems to work with vmprof from master:

pip install git+https://github.com/vmprof/vmprof-python.git#egg=vmprof
pip install -e .
crossbar version

(cpy310_1) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ crossbar version

    :::::::::::::::::
          :::::          _____                 __              _____  __
    :::::   :   :::::   / ___/______  ___ ___ / /  ___ _____  / __/ |/_/
    :::::::   :::::::  / /__/ __/ _ \(_-<(_-</ _ \/ _ `/ __/ / _/_>  <
    :::::   :   :::::  \___/_/  \___/___/___/_.__/\_,_/_/   /_/ /_/|_|
          :::::
    :::::::::::::::::   Crossbar.io FX v21.11.1 [19000101-0000000]

    Copyright (c) 2013-2021 Crossbar.io Technologies GmbH. All rights reserved.

 Crossbar.io        : 21.11.1
   txaio            : 21.2.1
   Autobahn         : 21.11.1
     UTF8 Validator : wsaccel-0.6.2
     XOR Masker     : wsaccel-0.6.2
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-1.0.2
     CBOR Codec     : cbor-1.0.0
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-2.0
   Twisted          : 21.7.0-EPollReactor
   LMDB             : 1.2.1/lmdb-0.9.29
   Python           : 3.10.0/CPython
   PIP              : 21.3.1
   NumPy            : 1.21.4
   zLMDB            : 21.2.1
   CFXDB            : 21.2.1
   XBR              : 21.2.1
 Frozen executable  : no
 Operating system   : Linux-5.4.0-90-generic-x86_64-with-glibc2.31
 Host machine       : x86_64
 Release key        : RWT8YYDwXKYOjxxrQbsnUGFiIMKO5MDuAt+0i0s12fxM3zpfy3TCP4rU

and starting a node

(cpy310_1) oberstet@intel-nuci7:~/scm/oberstet/wmc/node$ crossbar start

2021-11-24T13:14:00+0100 [Controller  61240] 
2021-11-24T13:14:00+0100 [Controller  61240]     :::::::::::::::::
2021-11-24T13:14:00+0100 [Controller  61240]           :::::          _____                      __
2021-11-24T13:14:00+0100 [Controller  61240]     :::::   :   :::::   / ___/____ ___   ___  ___  / /  ___ _ ____
2021-11-24T13:14:00+0100 [Controller  61240]     :::::::   :::::::  / /__ / __// _ \ (_-< (_-< / _ \/ _ `// __/
2021-11-24T13:14:00+0100 [Controller  61240]     :::::   :   :::::  \___//_/   \___//___//___//_.__/\_,_//_/
2021-11-24T13:14:00+0100 [Controller  61240]           :::::
2021-11-24T13:14:00+0100 [Controller  61240]     :::::::::::::::::   Crossbar v21.11.1
2021-11-24T13:14:00+0100 [Controller  61240] 
2021-11-24T13:14:00+0100 [Controller  61240]     Copyright (c) 2013-2021 Crossbar.io Technologies GmbH, licensed under AGPL 3.0.
2021-11-24T13:14:00+0100 [Controller  61240] 
2021-11-24T13:14:00+0100 [Controller  61240] Booting standalone node .. <crossbar.node.main._run_command_start>
2021-11-24T13:14:00+0100 [Controller  61240] Node configuration loaded [config_source=localfile, config_path=/home/oberstet/scm/oberstet/wmc/node/.crossbar/config.json]
2021-11-24T13:14:00+0100 [Controller  61240] Node key files exist and are valid. Node public key is 0xc98180f9e67557f69bb8d37bdb50a60e238525addbbbd4fe77231373110551f6
2021-11-24T13:14:00+0100 [Controller  61240] Node key loaded from /home/oberstet/scm/oberstet/wmc/node/.crossbar/key.priv
2021-11-24T13:14:00+0100 [Controller  61240] Entering event reactor ...
2021-11-24T13:14:00+0100 [Controller  61240] Starting node .. [<crossbar.node.node.Node.start>]

so the only issue left is vmprof (master works, but we need to be able to refer to a vmprof release in CB)"

om26er commented 2 years ago

so it seems to work with vmprof from master:

This is gold. Having to use Crossbar with CPython 3.10 and PyPy 3.8 is going to be a great.

oberstet commented 2 years ago

alright, let's see if we get any feedback https://github.com/vmprof/vmprof-python/issues/240 soonish. if not, we'll remove vmprof again .... actually, would love to keep it in CB, and the right thing to do would help with https://github.com/vmprof/vmprof-python/issues/241 .. yes yes ..

om26er commented 2 years ago

hmm, we could probably ship the new release with vmprof from master, I guess that could work as well ?

oberstet commented 2 years ago

we could probably ship the new release with vmprof from master, I guess that could work as well ?

yeah, changed it in the PR .. let's see what happens

we also already use own forks for 2 packages .. all of which is an annoyance, because there is no "stable wheel base" to do all the binaries building

because of above, and for repeatability, I wanted to:

  1. add a wheel building step to the CI here that uploads to our server (as a wheel base store to use subsequently)
  2. then, during subsequent CI steps, and for all the binaries building in particular, install CB from 1. (not PyPI)

not sure what's the status of above ... I think it might already work like this? anyways: that would solve the hassles IMO ..

oberstet commented 2 years ago

I've merged this now with using vmprof master .. we can wait a couple of days, for possibly vmprof doind a release, and also to add whatever smaller fixes or what still would accumulate before a release

oberstet commented 1 year ago

cpy 3.11 build fails with

      building '_vmprof' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/libbacktrace
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.11 -c src/_vmprof.c -o build/temp.linux-x86_64-cpython-311/src/_vmprof.o -Wno-unused -DVMPROF_LINUX=1 -DVMPROF_UNIX=1 "-I src/" "-I src/libbacktrace"
      src/_vmprof.c: In function ‘sample_stack_now’:
      src/_vmprof.c:301:53: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
        301 |     entry_count = vmp_walk_and_record_stack(tstate->frame, m, SINGLE_BUF_SIZE/sizeof(void*)-1, (int)skip, 0);
            |                                                     ^~~~~
            |                                                     cframe
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for vmprof
  Running setup.py clean for vmprof
Failed to build vmprof