Closed om26er closed 2 years ago
Here is a super basic crossbar config for a proxy worker, enough to reproduce the bug described above
{
"version": 2,
"workers": [
{
"type": "proxy",
"connections": {
"conn1": {
"transport": {
"type": "rawsocket",
"endpoint": {
"type": "tcp",
"host": "127.0.0.1",
"port": 9001
},
"url": "ws://localhost",
"serializer": "cbor"
}
}
},
"routes": {
"realm1": {
"system": "conn1",
"admin": "conn1"
}
},
"transports": [
{
"type": "universal",
"endpoint": {"type": "tcp", "port": 8080},
"rawsocket": {
"serializers": ["json", "cbor"],
"auth": {
"anonymous": {
"type": "static",
"role": "admin"
}
}
}
}
]
}
]
}
Since the code runs as a roundrobin, it "succeeds" on first attempt and fails on the other
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
Here are the relevant Crossbar logs. NOTE: the connection refused is expected as I am not running a backend.
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43348
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<N75Y-P4SH-XKC3-9R3V-MLMQ-W34P>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:16+0500 [Proxy 199315] realm1 admin
2022-03-26T02:27:16+0500 [Proxy 199315]
Traceback (most recent call last):
--- <exception caught here> ---
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1326, in map_backend
backend_proto = yield make_backend_connection(backend_config, frontend, self._cbdir)
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43350
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<APYW-FME9-NHTN-5P6F-E6SE-ESPR>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:16+0500 [Proxy 199315] realm1 admin
2022-03-26T02:27:16+0500 [Proxy 199315]
Traceback (most recent call last):
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 350, in _accept
backend_d = self._controller.map_backend(
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- <exception caught here> ---
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1309, in map_backend
backend_config = self.get_backend_config(realm, authrole)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1392, in get_backend_config
connection_id = route.config[role_name]
builtins.KeyError: 'admin'
2022-03-26T02:27:16+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:17+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43352
2022-03-26T02:27:17+0500 [Proxy 199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:17+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<FWXL-F9FV-PQRV-SJVH-FT65-LN6L>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:17+0500 [Proxy 199315] realm1 admin
2022-03-26T02:27:17+0500 [Proxy 199315]
Traceback (most recent call last):
--- <exception caught here> ---
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1326, in map_backend
backend_proto = yield make_backend_connection(backend_config, frontend, self._cbdir)
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
2022-03-26T02:27:17+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:18+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43354
2022-03-26T02:27:18+0500 [Proxy 199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:18+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<7W7V-PLHN-Q99L-QCVT-U7EK-H6LF>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:18+0500 [Proxy 199315] realm1 admin
2022-03-26T02:27:18+0500 [Proxy 199315]
Traceback (most recent call last):
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 350, in _accept
backend_d = self._controller.map_backend(
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- <exception caught here> ---
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
session = yield self._accept(hello_result)
File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1309, in map_backend
backend_config = self.get_backend_config(realm, authrole)
File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1392, in get_backend_config
connection_id = route.config[role_name]
builtins.KeyError: 'admin'
2022-03-26T02:27:18+0500 [Proxy 199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
Luckily this issue can be fixed with a one-liner
om26er@P1:~/scm/crossbario/crossbar$ git diff
diff --git a/crossbar/worker/proxy.py b/crossbar/worker/proxy.py
index 74869f73..11b07cf7 100644
--- a/crossbar/worker/proxy.py
+++ b/crossbar/worker/proxy.py
@@ -1384,6 +1384,7 @@ class ProxyController(TransportController):
assert self.has_role(realm_name, role_name)
routes = self._routes[realm_name]
+ routes = {route_id: route for route_id, route in routes.items() if role_name in route.config}
self._roundrobin_idx = (self._roundrobin_idx + 1) % len(routes)
route = list(routes.values())[self._roundrobin_idx]
If a realm route has only one role, everything works fine. However if there are multiple roles for a realm and the route is configured, the round robin logic is broken https://github.com/crossbario/crossbar/blob/fa0e418ea143efe6ab360fb5f438e5ebf46e8f15/crossbar/worker/proxy.py#L1386-L1388
In the current implementation, a new realm route is started for each role https://github.com/crossbario/crossbar/blob/fa0e418ea143efe6ab360fb5f438e5ebf46e8f15/crossbar/node/node.py#L962-L978
However, the round robin logic above doesn't check if a route has the specific role and tries to access it, resulting in a KeyError.