Update buildroot for crossbar v21.3.1

[ecorm]

The buildroot crossbar package does not work in the latest release/head of buildroot. After two days of playing dependency whac-a-mole, I've been able to make it work again with the following steps:

• Remove the package-crossbar/0002-Remove-idna-requirement.patch and replace it with another patch which does the following:
  • Change the idna requirement to >=2.5,<4 to match that of the Requests v2.27.1 package included with the latest buildroot. The idna v3.3 package is the one currently bundled with buildroot.
  • Change the urllib3 requirement to >=1.21.1,<1.27 to match that of the Requests v2.27.1 package included with the latest buildroot. The urllib3 v1.26.8 package is the one currently bundled with buildroot.
  • Remove the importlib-resources>=4.1.1 requirement. The importlib-resources 5.0 features have been merged into the CPython 3.10 included with the latest buildroot.
• Generate a new python-zlmdb package via utility/scanpypi. scanpypi fails before terminating, but it generates a viable python-zlmdb package nonetheless.
• Add a patch for the new python-zlmdb package which removes pytest-runner from the setup-requirements in setup.py.
• Bump buildroot's python-pyopenssl version to 21.0.0.
• Bump buildroot's python-snappy version to 0.6.0.
• Bump buildroot's python-trek version to 21.5.0.
• Add these missing dependencies in python-crossbar/Config.in: bcrypt, cffi, colorama, wsaccel, zlmdb, argon_cffi, cbor2, flatbuffers, msgpack, snappy, ujson

I will submit a patch upstream to the buildroot project with the above changes. I'll first need some time to familiarize myself with their submission conventions. I'll also make that patch available here for those who need it.

---

Upstream bug report: https://bugs.buildroot.org/show_bug.cgi?id=14556

[oberstet]

sounds like a concerted effort, cool!

let me add my perspective/experience gained over the years, maybe useful:

on the one hand, my personal view of distro packaging (that is, creating OS packages for every Python package) is: it necessarily leads to problems, as the Python package version that is repackaged at the OS level needs to "fit" any of the apps that are supposed to run on the system

in this context, I subsume under "OS" eg Debian packages, Yocto layers or Buildroot recipes

now, the alternative is to bundle Python packages needed by an app with the app

Python Virtualenv, Docker, snaps and PyInstaller self-contained EXEs are "bundling technologies"

this works a lot better (for me), but it still not working perfectly. the reason are "confluent dependencies" and the dependency resolution of pip.

Crossbar.io has ~100 direct and indirect Python package dependencies, and some of the packages are

• using lower and upper version bounds
• are used in 2 or more dependencies

this is the main source of issues remaining, and to prevent that, I've been working also with different upstreams and on our own packages to follow one rule:

only have lower version bounds in dependencies

the actual issue in pip is: it can run into an "infinite loop" when trying to figure out all possibilities for "allowed versions" with confluent dependencies and lower+upper bounds.

---

In our own dependencies, this rule is almost followed completely (it is not followed in all dependencies of course ..):

(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ grep "<" requirements-min.txt
# twisted/http2 needs <4:
h2>=3.2.0,<4.0.0
# twisted/http2 needs <6.0.0:
hyperframe>=5.2.0,<6.0.0
idna<2.6,>=2.5
urllib3>=1.21.1,<1.27

• h2 and hyperframe: we should look at Twisted if that is still required (the upper bound)
• idna and urllib: these are a constant PITA as you've already found out;) and the fact that requests does have upper bounds (https://github.com/psf/requests/blob/24dfbfde552bb8ce48eec415714cbe793cfa95bd/setup.py#L43), and requests is used in a lot of packages in turns makes this so annoying ...

[oberstet]

The rule "only have lower version bounds in dependencies" would be good IMO for both strategies: "OS" and "bundling techs".

For completeness, Crossbar.io also in general follow a second rule: "bump all dependencies upon own new release"

This 2nd rule is not a problem at all with approach "bundling techs".

It can lead to issues with the "OS" approach, as it means, a new OS level package version is required when we bump the lower version bound in our dep.

And this last piece is where a conflict of goals exists hence:

A) bump all deps upon release, use the "new/better deps": not a problem, because of the "bundling" approach for packaging Crossbar.io B) with "OS" approach, bumping deps in apps can lead to issues ..

now, I don't want to held A) "hostage" to the (bad) consequences of B), hence my usual reaction is: not my problem, update the OS package;) This isn't meant in an arrogant way, it is simply a logical conclusion of not willing to pay the price rgd the former ("hostage") ..

[ecorm]

Virtualenv worked well for us in the past for development, but we are moving away from deploying a Linux distro (currently Debian) on our embedded device. The embedded box that our hardware/software runs on is single-purpose and is not intended to be general-purpose like the way a hobbyist would experiment with a Raspberry Pi. When we perform remote updates, we want to update the entire firmware image all at once, including the OS.

I'm finding out that buildroot is ideal for our purposes, and it's been well worth my time getting Crossbar to work on it. Crossbar is the only Python app we need to run, so I'm free to tailor the buildroot Python packages as needed for Crossbar.

We'll also be migrating away from Crossbar due to it's large footprint (larger than the Linux Kernel!) and it's very slow startup time. While this is probably not an issue for a server/cloud platform, it is a serious issue for an embedded IoT device like ours.

We have live systems that need to keep running, so I still need Crossbar while we work on an alternative that better fits our needs.

[ecorm]

@oberstet , what is the rationale for Crossbar imposing versions for indirect dependencies?

[oberstet]

very slow startup time

you might be interested in

https://github.com/crossbario/crossbar/runs/4951083929?check_suite_focus=true#step:7:9 https://github.com/crossbario/crossbar/blob/master/test/test_imports.py

@om26er is also interested in cutting that startup time down. most of startup time is spent in imports.

the other factor (still need to add tests for that): since crossbar is a multi-process server, it will spawn new Pythons, and hence the way in which Python is built and linked matters. aka "shared". at least rgd memory.

what is the rationale for Crossbar imposing versions for indirect dependencies?

very good question! ;) reason is to work around the pip dependency resolution issues. when such an issue pops up (an indirect dependency of two or more direct dependencies etc), using an "artificial" direct dependency to pin the version needed solves the problem. finding out about all this and coming up with this working and semi-ugly approach took me many hours;) I absolutely hate this stuff. It is pointless, there isn't anything real to learn and it take hours and hours digging .. anyways. rant over;)

[ecorm]

Output of crossbar version on my BeagleBone Black device:

    :::::::::::::::::
          :::::          _____                      __
    :::::   :   :::::   / ___/____ ___   ___  ___  / /  ___ _ ____
    :::::::   :::::::  / /__ / __// _ \ (_-< (_-< / _ \/ _ `// __/
    :::::   :   :::::  \___//_/   \___//___//___//_.__/\_,_//_/
          :::::
    :::::::::::::::::   Crossbar v21.3.1

    Copyright (c) 2013-2000 Crossbar.io Technologies GmbH, licensed under AGPL 3.0.

 Crossbar.io        : 21.3.1
   txaio            : 21.2.1
   Autobahn         : 21.11.1
     UTF8 Validator : wsaccel-0.6.2
     XOR Masker     : wsaccel-0.6.2
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-1.0.3
     CBOR Codec     : cbor-1.0.0
     UBJSON Codec   : ubjson-0.16.1
     FlatBuffers    : flatbuffers-20220128024816
   Twisted          : 20.3.0-EPollReactor
   LMDB             : 1.2.1/lmdb-0.9.29
   Python           : 3.10.1/CPython
   PIP              : 
 Frozen executable  : no
 Operating system   : Linux-5.10.90-armv7l-with-glibc2.33
 Host machine       : armv7l
 Release key        : RWT906S6dtgOY3NZrHd3uFtXpAPFUc6+bdpADYJRJi4daUaCT5U/0OrO

[ecorm]

@oberstet If it's not too much trouble, could you or one of your colleagues confirm that the below Crossbar 21.3.1 dependencies are indirect?

attrs>=17.2.0 # via autobahn[twisted]
bcrypt>=3.1.6 # via twisted[conch]
constantly>=15.1.0 # via twisted
h2>=3.2.0 # via twisted[http2]
idna<2.6,>=2.5 # via twisted[tls]
incremental>=17.5.0 # via treq
netaddr>=0.7.19
priority>=1.3.0 # via twisted[http2]
py-ubjson>=0.9.0 # via autobahn[serialization]
service_identity>=17.0.0 # via autobahn[encryption]
u-msgpack-python>=2.4.1  # via autobahn[serialization]
urllib3<1.25,>=1.21.1 # via treq -> requests
wsaccel>=0.6.2; platform_python_implementation=='CPython' #via autobahn

I wasn't able to track down the dependency chain that leads to the netaddr package.

buildroot already pegs the versions of Python packages, so there's no point in "tricking" the build system under buildroot. Knowing which of the dependencies are indirect, I can then apply a patch that removes them from requirements-min.txt for the buildroot build.

[oberstet]

I wasn't able to track down the dependency chain that leads to the netaddr package.

I think netaddr comes via web3 - and only that? if so, it is an indirect dependency, but it isn't a confluent dependency, and thus shouldn't need to be present in crossbar's requirements:

(cpy39_3) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ pipdeptree --reverse --packages netaddr
netaddr==0.8.0
  - crossbar==21.11.1 [requires: netaddr>=0.8.0]
  - multiaddr==0.0.9 [requires: netaddr]
    - ipfshttpclient==0.8.0a2 [requires: multiaddr>=0.0.7]
      - web3==5.26.0 [requires: ipfshttpclient==0.8.0a2]
        - crossbar==21.11.1 [requires: web3>=5.13.1]
(cpy39_3) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ pipdeptree --packages crossbar
...
(cpy39_3) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ pipdeptree --packages crossbar --graph-output pdf > deps.pdf
...
(cpy39_3) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ pipdeptree --reverse --packages attrs
...

buildroot already pegs the versions of Python packages, so there's no point in "tricking" the build system under buildroot.

yes, and it doesn't hurt either, so why bother patching the requirements file?

[om26er]

One thing I'd like to add is that the latest crossbar is 21.11.1 even though it's not available on pypi.

@ecorm separately, I have recently packaged crossbar for Yocto (dunfell), it took a while but seems to have eventually worked. In my cases, I had to override a few Yocto recipes since their existing versions were older than what Crossbar needed. In your case it seems like you are using the "standalone" version of crossbar, the one that doesn't contains the newly open-sourced crossbarfx.

[om26er]

Regarding Crossbar startup performance: That's a concern that I want to address as well, since the startup time can be as much as 12 seconds (standalone personality) or 20 seconds (edge personality) on RaspberryPi 4 when running a single worker.

[oberstet]

One thing I'd like to add is that the latest crossbar is 21.11.1 even though it's not available on pypi.

yeah, sorry, I am aware - fwiw, this should happen soon after the transfer to the WAMP DAO ...

[oberstet]

Regarding Crossbar startup performance:

I think there could be quite a chunk of time to save with limited effort. my top list to first address would be:

1. bisect down the origin of import time spent via refining https://github.com/crossbario/crossbar/blob/master/test/test_imports.py
2. use a python binary built "shared"

then, optimize real world wall time from node start to 1st router worker started, listening and accepting wamp connections. a measuring program to measure this time should:

• remember walltime T0
• spawn crossbar CLI with defined config (1 router worker)
• try to wamp connect every 100ms in and endless loop until success
• compute walltime since T0

---

realistic startup time [RST] the point of using above 1 node with 1 router worker is: "startup time" can be well defined in application terms. it is the time until the node is actually usable for a WAMP client (as a WAMP router).

otherwise, with a node in the general case, how do you define "startup time" exactly?

minimal startup time [MST] the minimal startup time is the time from CLI start to controller process ready and beginning to process a local node config, or having registered its remote management API with the master node

import time [IT] the import time is the walltime for only importing all classes that the node controller / CLI will import when actually starting.

[ecorm]

buildroot already pegs the versions of Python packages, so there's no point in "tricking" the build system under buildroot.

yes, and it doesn't hurt either, so why bother patching the requirements file?

My motivation was to save buildroot maintenance headaches in the future, when the direct dependencies are updated and they decide to change their dependency requirements, which could then conflict with your indirect dependency requirements.

Upon further thought, it's only the indirect dependencies with upper version limits that are causing trouble (namely idna and urllib3). I can indeed leave the other ones alone.

One thing I'd like to add is that the latest crossbar is 21.11.1 even though it's not available on pypi.

I'm guessing the previous python-crossbar buildroot maintainer went with the latest tagged version on the Crossbar Github repo, which is still 21.3.1 as of today. We don't need the crossbarfx stuff, so it doesn't really bother me personally (unless there are important bugfixes that would affect us).

[ecorm]

If there are no objections, I'm patching* the Crossbar requirements-min.txt for buildroot as follows:

• idna>=2.5 (remove upper limit)
• urllib3>1.24.2 (remove upper limit and avoid CVE-2019-11324 vulnerability)
• Remove the importlib-resources requirement entirely as its features are merged in the CPython bundled with buildroot. There is no python-importlib-resources package available in buildroot, so removing this unneeded requirement is simpler than introducing a new Python package into buildroot.

(*) To be clear, by "patching", I mean having buildroot apply a patch file during the build process for its python-crossbar package.

[oberstet]

the previous python-crossbar buildroot maintainer

who is that? is there a link to that? my main worry is users expecting "python-crossbar buildroot " to work "as like crossbar" because it is named as such ...

which could then conflict with your indirect dependency requirements.

which would be the right thing. crossbar is defining it's dependencies based on crossbar requirements, and "what XYZ ships" should not be relevant in this decision.

[oberstet]

Upon further thought, it's only the indirect dependencies with upper version limits that are causing trouble (namely idna and urllib3). I can indeed leave the other ones alone.

yes, I agree, the upper limits is the annoying bit, and removing them would be great also here in this repo!

if it doesn't break stuff. now that we have a "lot if CI", we could dare to give it a try. but we need to at least also test our binaries repo which bakes EXE, snap, Docker, ..

Add a patch for the new python-zlmdb package which removes pytest-runner from the setup-requirements in setup.py.

this also sounds like it makes sense for zlmdb itself! I have no idea why it has "pytest-runner" in it's (non dev) regular requirements ..

actually: https://github.com/crossbario/zlmdb/issues/35#issuecomment-776351152

[oberstet]

Remove the importlib-resources requirement entirely as its features are merged in the CPython bundled with buildroot. There is no python-importlib-resources package available in buildroot

so it ships an incompatible/non-standard python. I am sure there will be a reason for this, but it is also the kinda reason I don't rely on "distro pythons" if in doubt;) anyways, you know what you are doing.

[oberstet]

ok, I checked the deps with upper bounds: https://github.com/crossbario/crossbar/issues/1937#issuecomment-1024585677

[ecorm]

@oberstet I just want to say that I understand your time is finite and you can't be expected to support every platform out there that deviates from your normal deployment procedures. I therefore appreciate the time you're giving in this discussion.

I can't explain why buildroot does the Python stuff the way it does, as I'm still pretty new to buildroot and I don't pretend to be a Python expert. The only thing I can say is that buildroot intentionally does not want to be a Linux distribution with a runtime package management system. Everything has to be cross-compiled on a host computer. Their approach makes sense for commercial embedded products that shouldn't be tampered with by end-users (think network routers and TV set-top boxes).

Please note that I'm not against tinkerers installing their own custom firmware in commercial products (e.g OpenWrt on routers), but it's usually understood that commercial vendors are not expected to provide any support or warranty if customers break their products by tampering with the firmware.

A WAMP router that can be natively cross-compiled would be a better fit for our situation, but I can't find any that has all the features we need (most importantly dynamic authentication/authorization).

[ecorm]

my main worry is users expecting "python-crossbar buildroot " to work "as like crossbar" because it is named as such ...

If it's any comfort, I'm bumping up the buildroot Python package versions as needed to meet the requirements of Crossbar and its dependencies.

[oberstet]

@ecorm alright, first of all: thanks a lot for your understanding! =) much appreciated

then, on the subject matter, I have added changes as discussed to autobahn and zlmdb:

• https://github.com/crossbario/zlmdb/pull/40
• https://github.com/crossbario/autobahn-python/pull/1514
• https://github.com/crossbario/autobahn-python/tree/v22.1.1
• https://github.com/crossbario/zlmdb/tree/v22.1.2
• https://pypi.org/project/autobahn/22.1.1/
• https://pypi.org/project/zlmdb/22.1.2/

now it's crossbar:

• https://github.com/crossbario/crossbar/pull/1938
• => v22.1.1 upcoming ...

[ecorm]

crossbar ... v22.1.1 upcoming

I'm going to take baby steps here and focus on making sure that Crossbar v21.3.1 works properly on buildroot.

If Crossbar v22.1.1 bumps a lot of its dependency requirements up to "the latest and greatest", it may take a Herculean effort to get all the corresponding buildroot Python packages updated to match. I'm not saying this to discourage you from bumping up your requirements; I'm just stating the reality of the situation.

[oberstet]

ok, so CI for crossbar v22.1.1 ran green, and that version now has tag on github.

publication to pypi failed though: https://gist.github.com/oberstet/073272467a126efa1cbc65e011921d8f

sorry, I am tired of this shit for today ..

[ecorm]

Fork incorporating my proposed fixes: https://github.com/ecorm/buildroot/tree/issue-14556-crossbar

[ecorm]

Unofficial pull request on my private buildroot fork to facilitate discussion before I submit a patch upstream: https://github.com/ecorm/buildroot/pull/1

Based on prior issues involving buildroot, this may be of interest to @mcondarelli , @tpetazzoni , @yegorich , @Evidlo

[oberstet]

note to self: the crossbar support in buildroot is here in upstream: https://github.com/buildroot/buildroot/tree/master/package/python-crossbar

---

@ecorm looks like you did a lot of hard work to add missing python packages, and also rgd some details! very cool. heroic=)

let me summarize the status after your PR would apply, well, "status" as I understand it.

1. upper version limits in dependencies

as rgd the annoying/bad upper limits on versions in dependencies of crossbar: we have now removed all but 3, and those remaining 3 are coming via Twisted (the upper limits replicate the Twisted limits exactly), and the need comes from the way pip handles dependency resolution with confluent dependencies.

IOW: we must keep these 3 limits for now, and removing them would require changes to Twisted or pip

2. autobahn dist flavors

it would be ideal to have autobahn[all] available on buildroot, or the flavors marked bold here:

• [x] asyncio: support asyncio as networking framework
• [x] twisted: support Twisted as networking framework
• [x] accelerate: make WebSocket faster on CPython (not PyPy)
• [x] compress: support WebSocket compression
• [x] serialization: support various WAMP serializers
• [x] encryption: support TLS and WAMP-cryptosign authentication
• [x] scram: support WAMP-scram authentication
• [ ] nvx: experimental intrinsics based WebSocket acceleration (do not use!)
• [ ] dev: only for Autobahn library development
• [ ] xbr: support WAMP end-to-end encryption; support Ethereum integration

if you like, pls tick which ones are supported in buildroot in above ^

crossbar requires autobahn[twisted,encryption,compress,serialization,scram,xbr]. but:

Remove the netaddr requirement, which is an indirect dependency via autobahn[xbr]->web3. buildroot's python-autobahn package currently does not support the autobahn[xbr] extras.

https://github.com/crossbario/autobahn-python/blob/dc626948d5ffbd741ec1e3a7906870eae1a4c69b/setup.py#L116

I would help rgd autobahn[xbr], as I have an interest in both WAMP end-to-end encryption, as well as Ethereum integration. This will allow independent operators of WAMP router nodes to interconnect their nodes and host WAMP realms in an decentralized way, while all user app traffic is end-to-end encrypted (and unreadable to node operators). This is the future - but that's my view;)

Anyways, if you want to enable autobahn[xbr] in buildroot, I would help, and this would provide the basis for above features in crossbar.

It will be annoying to "fix", likely even upstream changes. the Ethereum world is awesome, but fast moving, and Python support is still "developing" I'd say in some corners. let's fix it.

I should note: doing this work would generally profit buildroot and other Python stuff. eg the web3 package is pretty much always required for anything in this field.

my personal view: in some years everything will be directly or indirectly connected to blockchains (ethereum). it'll be like "before/after the Internet". no one will be asking about "why" anymore. supporting this in embedded devices is one crucial area.

[ecorm]

Thanks for your detailed response. I want to remind you that my work is only targeting v21.3.1 of Crossbar for now. It would be cool if we could also get the latest release working too, but let's please take it one step at a time. :-)

IOW: we must keep these 3 limits for now, and removing them would require changes to Twisted or pip

That's fine, we can just patch away the upper limits in buildroot (as needed) to compensate for the fact that it's not under a PIP environment.

if you like, pls tick which ones are supported in buildroot in above

My personal fork currently supports: twisted, accelerate, compress, serialization, encryption, and scram. (It won't let me check your boxes, sorry.)

asyncio

This is just an empty list in setup.py with the comment "backwards compatibility". Is this really needed?

dev

Wouldn't it be more likely that development takes place in a PIP virtualenv on a desktop or server environment? I don't think this would get much use in buildroot (thus you not bolding it, I presume). Supporting dev would require adding the following new Python packages in buildroot: pep8-namimg, flake8, pyflakes, twine, sphinx, sphinxcontrib-images, pyenchant, 'sphinxcontrib-spelling', sphinx_rtd_theme, awscli, qualname, wheel

xbr

This would require adding the following new Python packages in buildroot: xbr, web3, rlp, py-eth-sig-utils, py-ecc, eth-abi, mnemonic, base58, py-multihash

There is a scanpypi utility in buildroot that can generate a new python package by scanning PyPi, but it only seems to work on half of the packages I try, unfortunately.

[oberstet]

I want to remind you that my work is only targeting v21.3.1 of Crossbar for now. It would be cool if we could also get the latest release working too, but let's please take it one step at a time. :-)

yes, understood!

asyncio

indeed it is a noop, just there for backwards compat with previous autobahn versions that supported python 2

so we can consider it "checked" for buildroot

https://github.com/crossbario/autobahn-python/blob/7becdaae12db7abee390dd86d4e247f5071b91ab/setup.py#L275

dev

yes, you're right, definitely not needed on buildroot (the built image)

This would require adding the following new Python packages in buildroot: xbr, web3, rlp, py-eth-sig-utils, py-ecc, eth-abi, mnemonic, base58, py-multihash

thanks for verifying, this is a very good summary! of the work that would need to be done to get it working (obviously, I can be done .. just a matter of time/work)

---

I wanted to file an issue with buildroot for adding above python packages, so that autobahn[xbr] works, so that crossbar 20.3.1+ works.

they don't have an issue tracker on github? is there any, or maybe you can file it? I just want to set a marker and also save your list .. to come back to later and know what to work on right away ...

[ecorm]

I wanted to file an issue with buildroot for adding above python packages, so that autobahn[xbr] works, so that crossbar 20.3.1+ works.

buildroot's GitHub repo is a mirror only. The buildroot project was started in 2001, so it obviously predates GitHub. Here are links to their official servers:

• Official Git repo: https://git.busybox.net/buildroot/
• Official bug tracker: https://bugs.buildroot.org/
• Mailing list: http://lists.buildroot.org/mailman/listinfo/buildroot
• Patch submissions: https://patchwork.ozlabs.org/project/buildroot/list/

Before either of us files a feature request for the missing Python package needed for autobahn[xbr], please allow me to attempt generating them myself. I might get lucky with the scanpypi utility and only have a few Python packages I would need to write by hand. I'm going to start attempting it after I write this.

[ecorm]

It turns out the autobahn[xbr] dependencies is a deep rabbit hole that I would rather not get into. Every dependency spawns a bunch of other dependencies and I end up with a big tree graph of dependencies. web3 in particular has a bunch of upper limits in its dependency requirements that will cause me grief.

I've just discovered that buildroot already includes python-pip and python-virtualenv. Theoretically, one could build crossbar the "standard" way on the embedded device itself. The virtual environment could then be extracted as an archive for later deployment via buildroot (as a custom application). This archive would likely not be very portable, though. If the Python stuff links to C libraries, then a change in a C library's ABI would break things.

[oberstet]

.. is a deep rabbit hole that I would rather not get into

yes, I understand. it is already a challenge sometimes on a plain "standard" linux (and windows of course). and with both your ship (web3 et al) and your sea (the OS and run-time) moving all the same time, this is pretty hard. which is the reason I gave up on this approach altogether;) docker, snaps, static execs, ... that works. anyways, I am repeating myself;)

Theoretically, one could build crossbar the "standard" way on the embedded device itself.

I would not go there;) I have been there, it is absolutely terrible and a time black hole. just my 2cts.

---

I have done everything to support your effort, I won't spend time on porting the xbr deps for buildroot, you also not, so that means you are stuck on v20.3.1 :(

Just to be clear, the latest version of Crossbar.io is supported on embedded ARMv8 via either Docker or snaps (eg Ubuntu Core). I also know there is a Yocto layer for Crossbar.io (not sure if all the stuff is OSS). For Buildroot, under their approach of pre-built OS wide packages (incl. for Python), and with no post-image extensibility, neither Docker nor snaps help.

Can we close this issue?

[ecorm]

Theoretically, one could build crossbar the "standard" way on the embedded device itself.

I would not go there;) I have been there, it is absolutely terrible and a time black hole. just my 2cts.

I meant building it the way as described in Crossbar's Install for PyPy documentation. That worked well for us in the past on our embedded device when it was running Debian. Yes, it takes a long time to build, but once it's done, we can just copy the files to another device configured the same way.

I have done everything to support your effort, I won't spend time on porting the xbr deps for buildroot

I really appreciate your support, thank you. I didn't expect you to spend any time porting the xbr dependencies.

so that means you are stuck on v20.3.1

That's fine, as I've already mentioned that we'll be migrating away from Crossbar (and perhaps even WAMP) to something more "lean" for our simplistic usage. Crossbar is a fine "Swiss Army knife", but we really just need a spoon. :-) I have learned much in my involvement with WAMP, and don't regret it.

The work in my PR (with your kind support) will allow us to start migrating now to a buildroot-based deployment.

Can we close this issue?

Sure. There's not much more you can do, really. If you don't mind, I'll post updates here regarding my buildroot patch being accepted or rejected upstream. Worst case is that I keep my personal fork alive in case it helps others in a similar situation to ours.

[ecorm]

Theoretically, one could build crossbar the "standard" way on the embedded device itself. I meant building it the way as described in Crossbar's Install for PyPy documentation.

It turns out this is not possible due to buildroot's policy of not supporting a native compiler on the target. Yocto is amongst the alternatives they recommend if someone needs a complier on the target.

[ecorm]

I'm going to hold off submitting a patch upstream until I can confirm our app still runs with Crossbar on buildroot. This is going to take a while.

[ecorm]

Patch set submitted to buildroot on April 3: https://lists.buildroot.org/pipermail/buildroot/2022-April/640097.html

Only the python-snappy bump has been merged so far. There's a high volume of patch requests going on over there, and I wouldn't be surprised if the remainder of my patch set sits forgotten in limbo unless others express interest in seeing the patches go through.

It turns out that Crossbar v21.3 is not suitable for us because dynamic authorization (on which we depend) has quit working. I see the authorization RPCs being round tripped, but Crossbar doesn't proceed with forwarding the actual RPC after it was authorized.

It's possible this bug (if it is indeed on Crossbar's end) has been fixed in later versions, but it's unfeasible for me to deploy those later versions in Buildroot (as discussed previously, due to the XBR web of dependencies).

I don't want to trouble the Crossbar maintainers to fix this possible bug on an old revision of the code base, so I guess this is where I step off the Crossbar bus. I'm now working on adding router functionality to CppWAMP, which is something I wanted to do for a long while anyway.

Crossbar has been great for us as a rapid prototyping bridge between the frontend and backend of our app, please don't get me wrong. It's just that a native C++ router solution that can be embedded directly in our server app is more practical in terms of deployment, OS image size, and remote updates. I've considered wampcc, but it lacks some of the features we need. Bonefish seems no longer actively maintained.

[oberstet]

It turns out that Crossbar v21.3 is not suitable for us because dynamic authorization (on which we depend) has quit working.

fwiw, we've recently added integration testing specifically for authentication

https://github.com/crossbario/crossbar-examples/tree/master/authentication#test

I am currently expanding this testing for all auth methods (include wamp-scram and cookie), and for proxy-router worker configurations (where the authentication is done in proxy workers and forwarded to router workers)

https://github.com/crossbario/crossbar-examples/pull/148

I don't want to trouble the Crossbar maintainers to fix this possible bug on an old revision of the code base

from my point of view, the crucial thing we miss is the automated CI testing of all authentication features. this is a biggie, but we are working on it (above stuff). ah, and yes, we only have one branch and won't backport code. it's like chrome or FF .. "always latest is best". if we had more resources, we might do "release channels" some day ..

which brings me to: please let me ask again (possibly), I understand you can't use docker images of crossbar on your device, then can you use snaps? does snapd run on your device?

---

taking a step back, thinking about your wish for a simple single source self contained static router ..

the crossbar code base and everything could only evolve, and go through several phases of experiments with improving WAMP, and of larger changes to the code, by using a highly flexible light weight language like Python.

I have no doubts about that, and wouldn't see a better existing language for that.

however, now that WAMP and Crossbar.io have finally completed their end-of-design scope, it's time for stabilization. which is in fact what we do with Autobahn and Crossbar.io now. since the WAMP features are now defined and hashed out, one could of course dare to aim to reimplement even the router in a static, rigid language like C++

and if I could perform magic, I'd magically transpiled to Rust (not C++) ;)

[ecorm]

which brings me to: please let me ask again (possibly), I understand you can't use docker images of crossbar on your device, then can you use snaps? does snapd run on your device?

I don't want to use snapd in my device, even if it is available. I want the leanest possible OS image with all the useless fluff removed, and a natively compiled executable without a big runtime environment/interpreter. Think network router, or TV set-top box. Firmware updates should be in the hundreds of megabytes at most, not gigabytes. The typical end user will not have any control over the operating system configuration and will not be able to install any package like one would do on a server Linux distro.

Please understand that it's not some hobbyist project I'm working on, but something that will eventually be mass-produced (well, hopefully, that is).

It always was my plan to use Crossbar temporarily until a natively compiled solution could replace it. I'm just now forced to move on to the natively compiled solution now, and can't put it off any longer.

light weight language like Python

I'm sorry, but your meaning of "lightweight" must be different than mine. In terms of image size, Python support in buildroot is as large as the entire Linux kernel! I'm thinking you meant lightweight in terms of language complexity.

one could of course dare to aim to reimplement even the router in a static, rigid language like C++

That's up to you of course, but I don't design on promises so I have to assume that no one is going to write a complete C++ WAMP router for me for free just because I ask nicely. I think CppWAMP is already halfway to being a router. There are some rough edges I need smooth out before implementing router functionalty, but I don't doubt my ability to pull it off. At least with my own WAMP router implementaiton, I'll have full control over the code (except for networking and serialization libraries, but those are pretty stable/reliable). I'll be in a position to fix bugs myself and prioritize things the way I want.

and if I could perform magic, I'd magically transpiled to Rust (not C++) ;)

I'm keeping an eye on Rust, but I have decades of experience with C++ which makes it hard to abandon.

[oberstet]

That's up to you of course, but I don't design on promises so I have to assume that no one is going to write a complete C++ WAMP router for me for free just because I ask nicely.

well, it won't be me, that's for sure, as I think it's pointless. I might one day rewrite Crossbar.io in Rust - or not. anyways, if C++ and CppWAMP works for you, perfect! go with that!

[ecorm]

I want to make it clear that I'm not bashing on Crossbar. It's been very helpful as the initial bridge between the frontend and our embedded server app. It's just that it being a Python app is not compatible with my ultimate goal of having a very lean device image. The long startup time on an ARM device is also an issue for us, but that's probably insignificant for other situations (like on a server) where there's no expectation of a user interacting with a device right after turning it on.

[oberstet]

yes, I understand your choice, and that's totally fine!

It's just that it being a Python app is not compatible with my ultimate goal of having a very lean device image.

as laid out, I disagree, as it is objectively wrong, eg using snaps is light weight. but anyways, I don't care, use whatever works for you.

[om26er]

I don't want to use snapd in my device, even if it is available. I want the leanest possible OS image with all the useless fluff removed, and a natively compiled executable without a big runtime environment/interpreter.

May I suggest an alternative WAMP router out there, that is small in size (7.4 megabytes) and has very low memory footprint (2.3mb when started) and starts "instantly" even on raspberry pi. I am not sure about it's practicality in the cloud as I am not using it there but on embedded systems it's quite feasible IMO. It's Nexus, written in Go https://github.com/gammazero/nexus -- I basically think one should not be writing a WAMP router in cpp unless they really have to :-)

[ecorm]

@om26er Thanks for the suggestion. I remember checking it out and it lacked the dynamic authentication/authorization feature we need. And by "dynamic", I mean like Crossbar where it can call an external RPC to perform the authorization/authentication.

The ideal end goal for me is still a C++ router that I can embed directly in our C++ server app (the latter I will not rewriting in a different language). I don't want to have to interface with another library or executable written in a different language via clumsy C bindings or IPC.

Why does it seem like everyone is against me writing my own C++ router? Is anyone getting rich off of theirs? I'm more interested in selling our integrated product, not the middleware tech.

[oberstet]

Why does it seem like everyone is against me writing my own C++ router? Is anyone getting rich off of theirs?

for the record, I am not getting rich, and I will support you - from the sidelines (any spec stuff should it matter, or client interop) - but I don't personally feel motivated or convinced about specifically C++. the latter would be different for Rust, but again, this is really just how I personally feel, which of course shouldn't distract you;)