Regression with Twisted 21.2.0

[oberstet]

here is twisted v20.3.0 (working):

(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ pip install -U twisted
Collecting twisted
  Using cached https://files.pythonhosted.org/packages/f2/16/3eb9c66a7bfb5220c7bcbaaac33d359fe8a157b028959cd210983749b2e0/Twisted-21.2.0-py3-none-any.whl
Requirement already satisfied, skipping upgrade: attrs>=19.2.0 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (20.3.0)
Requirement already satisfied, skipping upgrade: incremental>=16.10.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (17.5.0)
Requirement already satisfied, skipping upgrade: constantly>=15.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (15.1.0)
Requirement already satisfied, skipping upgrade: Automat>=0.8.0 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (20.2.0)
Requirement already satisfied, skipping upgrade: zope.interface>=4.4.2 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (5.2.0)
Requirement already satisfied, skipping upgrade: hyperlink>=17.1.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted) (21.0.0)
Requirement already satisfied, skipping upgrade: six in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from Automat>=0.8.0->twisted) (1.15.0)
Requirement already satisfied, skipping upgrade: setuptools in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from zope.interface>=4.4.2->twisted) (52.0.0)
Requirement already satisfied, skipping upgrade: idna>=2.5 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from hyperlink>=17.1.1->twisted) (2.5)
Installing collected packages: twisted
  Found existing installation: Twisted 20.3.0
    Uninstalling Twisted-20.3.0:
      Successfully uninstalled Twisted-20.3.0
Successfully installed twisted-21.2.0
WARNING: You are using pip version 19.3.1; however, version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ pip show twisted
Name: Twisted
Version: 21.2.0
Summary: An asynchronous networking framework written in Python
Home-page: https://twistedmatrix.com/
Author: Twisted Matrix Laboratories
Author-email: twisted-python@twistedmatrix.com
License: MIT
Location: /home/oberstet/cpy391_1/lib/python3.9/site-packages
Requires: Automat, incremental, attrs, zope.interface, constantly, hyperlink
Required-by: txtorcon, treq
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ docker-compose up master
WARNING: Some networks were defined but are not used by any service: home1
Starting master ... done
Attaching to master
master        | 
master        | 2021-03-02T14:10:10+0000 [Controller      1] 
master        | 2021-03-02T14:10:10+0000 [Controller      1]     :::::::::::::::::
master        | 2021-03-02T14:10:10+0000 [Controller      1]           :::::          _____                 __              _____  __
master        | 2021-03-02T14:10:10+0000 [Controller      1]     :::::   :   :::::   / ___/______  ___ ___ / /  ___ _____  / __/ |/_/
master        | 2021-03-02T14:10:10+0000 [Controller      1]     :::::::   :::::::  / /__/ __/ _ \(_-<(_-</ _ \/ _ `/ __/ / _/_>  <
master        | 2021-03-02T14:10:10+0000 [Controller      1]     :::::   :   :::::  \___/_/  \___/___/___/_.__/\_,_/_/   /_/ /_/|_|
master        | 2021-03-02T14:10:10+0000 [Controller      1]           :::::
master        | 2021-03-02T14:10:10+0000 [Controller      1]     :::::::::::::::::   Crossbar.io FX v21.2.2.dev4 [00000000-0000000]
master        | 2021-03-02T14:10:10+0000 [Controller      1] 
master        | 2021-03-02T14:10:10+0000 [Controller      1]     Copyright (c) 2013-2021 Crossbar.io Technologies GmbH. All rights reserved.
master        | 2021-03-02T14:10:10+0000 [Controller      1] 
master        | 2021-03-02T14:10:10+0000 [Controller      1] Booting master node .. <crossbar.node.main._run_command_start>
master        | 2021-03-02T14:10:10+0000 [Controller      1] Node key files exist and are valid. Node public key is 0x036e03643e1794e5f3e4cff7535a9dd5e887a855c2df48dc723bee0837273f7a
master        | 2021-03-02T14:10:10+0000 [Controller      1] Node key loaded from /master/.crossbar/key.priv
master        | 2021-03-02T14:10:10+0000 [Controller      1] Node configuration loaded [config_source=default, config_path=/usr/local/lib/python3.9/site-packages/crossbarfx/master/node/config.json]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Entering event reactor ...
master        | 2021-03-02T14:10:10+0000 [Controller      1] Starting node (initialize master-node personality) .. [<crossbarfx.master.node.node.FabricCenterNode.start>]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Starting node (initialize edge-node personality) .. [<crossbarfx.edge.node.node.FabricNode.start>]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Starting node .. [<crossbar.node.node.Node.start>]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Node ID center1 set from explicit configuration
master        | 2021-03-02T14:10:10+0000 [Controller      1] RouterFactory.start_realm: router created for realm "crossbar"
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbar.node.node.Node._add_global_roles> node-wide role "controller" added on node management router realm "crossbar"
master        | 2021-03-02T14:10:10+0000 [Controller      1] attached session 7173242369758155 to realm "crossbar" (authid="serviceagent", authrole="trusted") <crossbar.router.router.Router.attach>
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbar.router.service.RouterServiceAgent.onJoin>: realm service session attached to realm "crossbar" [session_id=7173242369758155, authid="serviceagent", authrole="trusted", on_ready=None]
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.start> router service agent session attached [<crossbar.router.service.RouterServiceAgent>]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Docker daemon integration disabled!
master        | 2021-03-02T14:10:10+0000 [Controller      1] attached session 1250832647707903 to realm "crossbar" (authid="nodecontroller", authrole="controller") <crossbar.router.router.Router.attach>
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbar.node.controller.NodeController.onJoin>: joined realm="crossbar" on local node management router [authid="nodecontroller", authrole="controller"]
master        | 2021-03-02T14:10:10+0000 [Controller      1] Signal handler installed on process 1 thread 140235421562688
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.start> node controller session attached [<crossbarfx.edge.node.node.FabricNodeControllerSession>]
master        | 2021-03-02T14:10:10+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.boot>::NODE_BOOT_BEGIN
master        | 2021-03-02T14:10:10+0000 [Controller      1] Booting node <crossbarfx.master.node.node.FabricCenterNode.boot>
master        | 2021-03-02T14:10:10+0000 [Controller      1] Booting node from local configuration [parallel_worker_start=False] .. <crossbar.node.node.Node.boot_from_config>
master        | 2021-03-02T14:10:10+0000 [Controller      1] Will start 2 workers ..
master        | 2021-03-02T14:10:10+0000 [Controller      1] Order node to start "Router cfrouter1" ..
master        | 2021-03-02T14:10:10+0000 [Controller      1] Starting router-worker "cfrouter1" .. <crossbar.node.controller.NodeController.start_worker>
master        | 2021-03-02T14:10:10+0000 [Controller      1] Unhandled Error
master        | Traceback (most recent call last):
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/controller.py", line 377, in start_worker
master        |     return self._start_native_worker(worker_type, worker_id, worker_options, details=details)
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/controller.py", line 725, in _start_native_worker
master        |     d = ep.connect(transport_factory)
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/common/twisted/processutil.py", line 112, in connect
master        |     self._spawnProcess(wrapped,
master        |   File "/extra/site-packages/twisted/internet/posixbase.py", line 397, in spawnProcess
master        |     return process.Process(
master        | --- <exception caught here> ---
master        |   File "/extra/site-packages/twisted/internet/process.py", line 732, in __init__
master        |     self.proto.makeConnection(self)
master        |   File "/extra/site-packages/twisted/internet/endpoints.py", line 365, in makeConnection
master        |     return self.protocol.makeConnection(self.transport)
master        |   File "/extra/site-packages/twisted/internet/protocol.py", line 508, in makeConnection
master        |     self.connectionMade()
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/native.py", line 49, in connectionMade
master        |     WampWebSocketClientProtocol.connectionMade(self)
master        |   File "/usr/local/lib/python3.9/site-packages/autobahn/twisted/websocket.py", line 244, in connectionMade
master        |     self.peer = peer2str(self.transport.getPeer())
master        |   File "/extra/site-packages/twisted/internet/process.py", line 946, in getPeer
master        |     raise NotImplementedError()
master        | builtins.NotImplementedError: 
master        | 
master        | 2021-03-02T14:10:10+0000 [Controller      1] worker-specific role "crossbar.worker.cfrouter1" added on node management router realm "crossbar" <crossbarfx.master.node.node.FabricCenterNode._add_worker_role>
master        | 
^CGracefully stopping... (press Ctrl+C again to force)
Stopping master ... done
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ 

here is twisted v21.2.0

(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ pip install -U "twisted==21.2.0"
Requirement already up-to-date: twisted==21.2.0 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (21.2.0)
Requirement already satisfied, skipping upgrade: constantly>=15.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (15.1.0)
Requirement already satisfied, skipping upgrade: zope.interface>=4.4.2 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (5.2.0)
Requirement already satisfied, skipping upgrade: hyperlink>=17.1.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (21.0.0)
Requirement already satisfied, skipping upgrade: Automat>=0.8.0 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (20.2.0)
Requirement already satisfied, skipping upgrade: attrs>=19.2.0 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (20.3.0)
Requirement already satisfied, skipping upgrade: incremental>=16.10.1 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from twisted==21.2.0) (17.5.0)
Requirement already satisfied, skipping upgrade: setuptools in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from zope.interface>=4.4.2->twisted==21.2.0) (52.0.0)
Requirement already satisfied, skipping upgrade: idna>=2.5 in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from hyperlink>=17.1.1->twisted==21.2.0) (2.5)
Requirement already satisfied, skipping upgrade: six in /home/oberstet/cpy391_1/lib/python3.9/site-packages (from Automat>=0.8.0->twisted==21.2.0) (1.15.0)
WARNING: You are using pip version 19.3.1; however, version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ pip show twisted
Name: Twisted
Version: 21.2.0
Summary: An asynchronous networking framework written in Python
Home-page: https://twistedmatrix.com/
Author: Twisted Matrix Laboratories
Author-email: twisted-python@twistedmatrix.com
License: MIT
Location: /home/oberstet/cpy391_1/lib/python3.9/site-packages
Requires: zope.interface, constantly, incremental, attrs, Automat, hyperlink
Required-by: txtorcon, treq
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ docker-compose up master
WARNING: Some networks were defined but are not used by any service: home1
Starting master ... done
Attaching to master
master        | 
master        | 2021-03-02T14:14:29+0000 [Controller      1] 
master        | 2021-03-02T14:14:29+0000 [Controller      1]     :::::::::::::::::
master        | 2021-03-02T14:14:29+0000 [Controller      1]           :::::          _____                 __              _____  __
master        | 2021-03-02T14:14:29+0000 [Controller      1]     :::::   :   :::::   / ___/______  ___ ___ / /  ___ _____  / __/ |/_/
master        | 2021-03-02T14:14:29+0000 [Controller      1]     :::::::   :::::::  / /__/ __/ _ \(_-<(_-</ _ \/ _ `/ __/ / _/_>  <
master        | 2021-03-02T14:14:29+0000 [Controller      1]     :::::   :   :::::  \___/_/  \___/___/___/_.__/\_,_/_/   /_/ /_/|_|
master        | 2021-03-02T14:14:29+0000 [Controller      1]           :::::
master        | 2021-03-02T14:14:29+0000 [Controller      1]     :::::::::::::::::   Crossbar.io FX v21.2.2.dev4 [00000000-0000000]
master        | 2021-03-02T14:14:29+0000 [Controller      1] 
master        | 2021-03-02T14:14:29+0000 [Controller      1]     Copyright (c) 2013-2021 Crossbar.io Technologies GmbH. All rights reserved.
master        | 2021-03-02T14:14:29+0000 [Controller      1] 
master        | 2021-03-02T14:14:29+0000 [Controller      1] Booting master node .. <crossbar.node.main._run_command_start>
master        | 2021-03-02T14:14:29+0000 [Controller      1] Node key files exist and are valid. Node public key is 0x036e03643e1794e5f3e4cff7535a9dd5e887a855c2df48dc723bee0837273f7a
master        | 2021-03-02T14:14:29+0000 [Controller      1] Node key loaded from /master/.crossbar/key.priv
master        | 2021-03-02T14:14:29+0000 [Controller      1] Node configuration loaded [config_source=default, config_path=/usr/local/lib/python3.9/site-packages/crossbarfx/master/node/config.json]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Entering event reactor ...
master        | 2021-03-02T14:14:29+0000 [Controller      1] Starting node (initialize master-node personality) .. [<crossbarfx.master.node.node.FabricCenterNode.start>]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Starting node (initialize edge-node personality) .. [<crossbarfx.edge.node.node.FabricNode.start>]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Starting node .. [<crossbar.node.node.Node.start>]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Node ID center1 set from explicit configuration
master        | 2021-03-02T14:14:29+0000 [Controller      1] RouterFactory.start_realm: router created for realm "crossbar"
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbar.node.node.Node._add_global_roles> node-wide role "controller" added on node management router realm "crossbar"
master        | 2021-03-02T14:14:29+0000 [Controller      1] attached session 3249540238640430 to realm "crossbar" (authid="serviceagent", authrole="trusted") <crossbar.router.router.Router.attach>
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbar.router.service.RouterServiceAgent.onJoin>: realm service session attached to realm "crossbar" [session_id=3249540238640430, authid="serviceagent", authrole="trusted", on_ready=None]
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.start> router service agent session attached [<crossbar.router.service.RouterServiceAgent>]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Docker daemon integration disabled!
master        | 2021-03-02T14:14:29+0000 [Controller      1] attached session 7788850473110698 to realm "crossbar" (authid="nodecontroller", authrole="controller") <crossbar.router.router.Router.attach>
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbar.node.controller.NodeController.onJoin>: joined realm="crossbar" on local node management router [authid="nodecontroller", authrole="controller"]
master        | 2021-03-02T14:14:29+0000 [Controller      1] Signal handler installed on process 1 thread 140614616364864
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.start> node controller session attached [<crossbarfx.edge.node.node.FabricNodeControllerSession>]
master        | 2021-03-02T14:14:29+0000 [Controller      1] <crossbarfx.master.node.node.FabricCenterNode.boot>::NODE_BOOT_BEGIN
master        | 2021-03-02T14:14:29+0000 [Controller      1] Booting node <crossbarfx.master.node.node.FabricCenterNode.boot>
master        | 2021-03-02T14:14:29+0000 [Controller      1] Booting node from local configuration [parallel_worker_start=False] .. <crossbar.node.node.Node.boot_from_config>
master        | 2021-03-02T14:14:29+0000 [Controller      1] Will start 2 workers ..
master        | 2021-03-02T14:14:29+0000 [Controller      1] Order node to start "Router cfrouter1" ..
master        | 2021-03-02T14:14:29+0000 [Controller      1] Starting router-worker "cfrouter1" .. <crossbar.node.controller.NodeController.start_worker>
master        | 2021-03-02T14:14:29+0000 [Controller      1] Unhandled Error
master        | Traceback (most recent call last):
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/controller.py", line 377, in start_worker
master        |     return self._start_native_worker(worker_type, worker_id, worker_options, details=details)
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/controller.py", line 725, in _start_native_worker
master        |     d = ep.connect(transport_factory)
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/common/twisted/processutil.py", line 112, in connect
master        |     self._spawnProcess(wrapped,
master        |   File "/extra/site-packages/twisted/internet/posixbase.py", line 397, in spawnProcess
master        |     return process.Process(
master        | --- <exception caught here> ---
master        |   File "/extra/site-packages/twisted/internet/process.py", line 732, in __init__
master        |     self.proto.makeConnection(self)
master        |   File "/extra/site-packages/twisted/internet/endpoints.py", line 365, in makeConnection
master        |     return self.protocol.makeConnection(self.transport)
master        |   File "/extra/site-packages/twisted/internet/protocol.py", line 508, in makeConnection
master        |     self.connectionMade()
master        |   File "/usr/local/lib/python3.9/site-packages/crossbar/node/native.py", line 49, in connectionMade
master        |     WampWebSocketClientProtocol.connectionMade(self)
master        |   File "/usr/local/lib/python3.9/site-packages/autobahn/twisted/websocket.py", line 244, in connectionMade
master        |     self.peer = peer2str(self.transport.getPeer())
master        |   File "/extra/site-packages/twisted/internet/process.py", line 946, in getPeer
master        |     raise NotImplementedError()
master        | builtins.NotImplementedError: 
master        | 
master        | 2021-03-02T14:14:29+0000 [Controller      1] worker-specific role "crossbar.worker.cfrouter1" added on node management router realm "crossbar" <crossbarfx.master.node.node.FabricCenterNode._add_worker_role>
master        | 
^CGracefully stopping... (press Ctrl+C again to force)
Stopping master ... done
(cpy391_1) oberstet@intel-nuci7:~/scm/s-things/thingcloud$ 

[oberstet]

awesome=) here is what happened:

• historically, twisted did not follow its own interfaces, and the process protocol implementation lacked "getPeer" altogether
• for that reasons, I am treating this by catching "AttributeError" https://github.com/crossbario/autobahn-python/blob/014cd9e4eabcff7ff56d7074c43865bf470b2aa7/autobahn/twisted/websocket.py#L245
• that worked fine until twisted decided to fix the original bug (lacking interface impl) 9 months ago: https://github.com/twisted/twisted/blame/trunk/src/twisted/internet/process.py https://github.com/twisted/twisted/commit/12d343033038e5e05abbed322b6c97bed7042cb5
• so the fixing of the bug in twisted made our workaround for the (no longer existent) bug break crossbar!

[oberstet]

the fix here should be a one liner:

except AttributeError, NotImplementedError:

[oberstet]

[oberstet]

this is fixed in

• https://github.com/crossbario/autobahn-python/pull/1469
• https://github.com/crossbario/autobahn-python/issues/1470