search

crossbario / crossbar

Crossbar.io - WAMP application router
https://crossbar.io/
Other
2.05k stars 274 forks source link

XBR extra on autobahn requirement require Cairo and PyGObject system dependencies to install #1955

Closed jacobanana closed 2 years ago

jacobanana commented 2 years ago

Hello,

I have recently updated to crossbar 22.2.1 from 21.3.1 and found multiple system dependencies were needed to be able to install crossbar.

On MacOS, it was necessary to install cairo and PyGObject using homebrew before being able to install crossbar

brew install cairo pygobject3

On Linux, those had to be installed with apt:

apt install libcairo2-dev libgirepository1.0-dev

PyGObject3 is a dependency of autobahn[xbr] only which is now installed by default with crossbar and seems to be useful for a very niche use case. This didn't use to be the case on 21.3.1, and I have noted a huge increase in the number of dependencies (probably unnecessary to most users, unless working on blockchain projects)

In addition, this created new dependency on vmprof which appears to be broken on most recent python versions (#1949). This meant that I had to stick to python 3.9.5 for this to build correctly on MacOS. I haven't found a solution for Ubuntu 18.04. The workaround proposed

Is XBR really necessary for the average crossbar user? Could this not be an extra for those who really need it? This would make crossbar more accessible to most.

Regards

Adrien

oberstet commented 2 years ago

This would make crossbar more accessible to most.

best for users is to rely on binary packages:

this way you get a self-contained deterministic software stack with only a single dependency: docker / snap

Could this not be an extra for those who really need it?

we don't have the resources to support the complexities arising from multiple installation flavors

Could this not be an extra for those who really need it?

we have recently open-sourced the multi-node (router mesh) code, which is now part of crossbar open-source. this is still "alpha", and one reason for the increase in deps lately ..

d-curtis commented 2 years ago

I have also encountered this issue, and it's keeping my project locked to Python 3.9.5 as a result of the dependency issues. I agree it seems unusual for a WAMP router to contain this functionality as standard, especially as it causes problems for the simplest use case.

best for users is to rely on binary packages

This is currently unclear, with the readme and docs directing users to get Crossbar from PyPI. There's a note in the docs about Docker, but the OSX install guide still points to installing through pip. I think the majority of users would rather use pip than setting up a Docker environment to build Crossbar each release.

I have not yet tested building from the Docker images - has this been seen to resolve the issues with upstream dependencies? It seems strange that the main line "stable" release includes dependencies that have been proven to not work with an otherwise fully up-to-date system.

oberstet commented 2 years ago

I have not yet tested building from the Docker images

you don't have to build anything: docker run --rm crossbario/crossbar:cpy-slim-amd64 version. and you can choose between cpython and pypy:

(base) oberstet@intel-nuci7:~$ docker run --rm crossbario/crossbar:cpy-slim-amd64 version

    :::::::::::::::::
          :::::          _____                 __              _
    :::::   :   :::::   / ___/______  ___ ___ / /  ___ _____  (_)__
    :::::::   :::::::  / /__/ __/ _ \(_-<(_-</ _ \/ _ `/ __/ / / _ \
    :::::   :   :::::  \___/_/  \___/___/___/_.__/\_,_/_/ (_)_/\___/
          :::::
    :::::::::::::::::   Crossbar.io v22.3.1.dev1 [19000101-0000000]

    Copyright (c) 2013-2022 Crossbar.io Technologies GmbH. Licensed under EUPLv1.2.

 Crossbar.io        : 22.3.1.dev1
   txaio            : 22.2.1
   Autobahn         : 22.2.2
     UTF8 Validator : wsaccel-0.6.2
     XOR Masker     : wsaccel-0.6.2
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-1.0.3
     CBOR Codec     : cbor2-5.4.2.post1
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-2.0
   Twisted          : 22.2.0-EPollReactor
   LMDB             : 1.3.0/lmdb-0.9.29
   Python           : 3.9.10/CPython
   PIP              : 22.0.3
   NumPy            : 1.22.2
   zLMDB            : 22.3.1.dev1
   CFXDB            : 22.2.2
   XBR              : 21.2.1
 Frozen executable  : no
 Operating system   : Linux-5.4.0-100-generic-x86_64-with-glibc2.31
 Host machine       : x86_64
 Release key        : RWRPe9BqdJL2vTjTj5h9evaVY4LiFHmbi8Gy1T83OVnWMFkYm9tQEOu4

(base) oberstet@intel-nuci7:~$ docker run --rm crossbario/crossbar:pypy-slim-amd64 version

    :::::::::::::::::
          :::::          _____                 __              _
    :::::   :   :::::   / ___/______  ___ ___ / /  ___ _____  (_)__
    :::::::   :::::::  / /__/ __/ _ \(_-<(_-</ _ \/ _ `/ __/ / / _ \
    :::::   :   :::::  \___/_/  \___/___/___/_.__/\_,_/_/ (_)_/\___/
          :::::
    :::::::::::::::::   Crossbar.io v22.3.1.dev1 [19000101-0000000]

    Copyright (c) 2013-2022 Crossbar.io Technologies GmbH. Licensed under EUPLv1.2.

 Crossbar.io        : 22.3.1.dev1
   txaio            : 22.2.1
   Autobahn         : 22.2.2
     UTF8 Validator : autobahn
     XOR Masker     : autobahn
     JSON Codec     : stdlib
     MsgPack Codec  : umsgpack-2.7.1
     CBOR Codec     : cbor2-5.4.2.post1
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-2.0
   Twisted          : 22.2.0-EPollReactor
   LMDB             : 1.3.0/lmdb-0.9.29
   Python           : 3.7.12/PyPy-7.3.8
   PIP              : 22.0.3
   NumPy            : 1.21.5
   zLMDB            : 22.3.1.dev1
   CFXDB            : 22.2.2
   XBR              : 21.2.1
 Frozen executable  : no
 Operating system   : Linux-5.4.0-100-generic-x86_64-with-debian-11.2
 Host machine       : x86_64
 Release key        : RWRPe9BqdJL2vTjTj5h9evaVY4LiFHmbi8Gy1T83OVnWMFkYm9tQEOu4

(base) oberstet@intel-nuci7:~$
oberstet commented 2 years ago

This is currently unclear, with the readme and docs directing users to get Crossbar from PyPI. There's a note in the docs about Docker, but the OSX install guide still points to installing through pip.

yes, this (docs) is sadly outdate. fwiw, yeah, sucks, and better docs would be nice. anyways, just a quick tip as I'm replying, what I usually do is use docker-compose with a .env file

# env vars defined here are already available during "docker-compose" phase (not only inside containers)
UID=1000
GID=1000

# Autobahn|Python image
#
# docker inspect --format='{{index .RepoDigests 0}}' crossbario/autobahn-python:cpy-amd64-22.2.1
#
DOCKER_IMAGE_AUTOBAHN_PYTHON=crossbario/autobahn-python@sha256:04b18ba89d48956ef1f2e474b9db06ebdd6e2609af633edd17c2650d659f5b81

# Crossbar.io image
#
# docker inspect --format='{{index .RepoDigests 0}}' crossbario/crossbar:cpy-slim-amd64-22.2.1
#
# DOCKER_IMAGE_CROSSBAR=crossbario/crossbar:cpy-slim-amd64-22.2.1@sha256:056a144d0f38f3231f2bc1541a1019b8b3c062828a4ce7ea934827f987f78c1b
DOCKER_IMAGE_CROSSBAR=crossbario/crossbar:pypy-slim-amd64-22.2.1@sha256:fb3a35335ea306b722aee02940faa8381811abb53e19cc2373ff8fcb71f796ff

so refer to exactly specified images

jacobanana commented 2 years ago

Hi Tobias,

Thanks for your reply. I understand most people will use this in a virtualised environment and that Docker is the way to go. Unfortunately, I can't use docker in my case and would like to deploy using old school virtualenv.

Is there a way of getting a lightweight version of crossbar? I've created a fork to try and slim things down, are you able to point me towards some help on how to run the unit tests?

Regards Adrien

oberstet commented 2 years ago

Is there a way of getting a lightweight version of crossbar?

docker is "lightweight" in my definition. there is only python package dist flavor if you mean that

I've created a fork to try and slim things down, are you able to point me towards some help on how to run the unit tests?

just type tox to run all tests. it's all in this repo, base on tox, pytest/trial and github actions:

you can also use a different router: https://wamp-proto.org/implementations.html#routers, eg https://github.com/gammazero/nexus is a nice one that is "lightweight" under some definition I'd say

oberstet commented 2 years ago

rereading, forgot to mention, rgd the specific deps you mention

libcairo2-dev libgirepository1.0-dev

this specific dependency comes from autobahn[xbr], which is a dependency of crossbar. and while crossbar uses some of the xbr stuff, it is not using anything that requires above.

so above two dependencies could be removed when we had autobahn[xbr] and split out autobahn[gobject] or similar, and then only kept the former dependency in crossbar

since we are using distribution flavors in autobahn (but not crossbar), this would fit in and I'd support such a change in general

oberstet commented 2 years ago

moved to https://github.com/crossbario/autobahn-python/issues/1532

oberstet commented 2 years ago

fixed via https://github.com/crossbario/autobahn-python/pull/1539