rlinks: existing procedures not forwarded

[asimfarooq5]

I have two crossbar instances. instance A is configured to connect to instance B using rlinks.

If the procedures are registered on instance A before the connection to instance B is established, it seems the invocations are not forwarded to instance B.

The only way to get this working is to establish a rlink connection between the two instances before registering the procedures.

[asimfarooq5]

configuration for instance A looks like

{
    "$schema": "https://raw.githubusercontent.com/crossbario/crossbar/master/crossbar.json",
    "version": 2,
    "controller": {
    },
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "name": "realm1",
                    "roles": [
                        {
                            "name": "anonymous",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": false,
                                        "publisher": false
                                    },
                                    "cache": true
                                }
                            ]
                        },
                        {
                            "name": "router2router",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": true,
                                        "publisher": true
                                    },
                                    "cache": true
                                }
                            ]
                        }
                    ],
                    "rlinks": [
                        {
                            "id": "edge",
                            "realm": "realm1",
                            "transport": {
                                "type": "rawsocket",
                                "endpoint": {
                                    "type": "tcp",
                                    "host": "localhost",
                                    "port": 6688
                                },
                                "url": "rs://localhost:6688",
                                "serializer": "cbor"
                            },
                            "forward_local_events": false,
                            "forward_remote_events": false,
                            "forward_local_invocations": true,
                            "forward_remote_invocations": true
                        }
                    ]
                }
            ],          
            "transports": [
                {
                    "type": "rawsocket",
                    "endpoint": {
                        "type": "tcp",
                        "port": 5588,
                        "backlog": 1024
                    },
                    "options": {
                        "max_message_size": 1048576
                    },
                    "serializers": [
                       "cbor",
                       "msgpack",
                       "json"
                   ]            
                },      
                {
                    "type": "web",
                    "endpoint": {
                        "type": "tcp",
                        "port": 8080,
                        "backlog": 1024
                    },
                    "paths": {
                        "ws": {
                            "type": "websocket",
                            "serializers": [
                                "cbor", "msgpack", "json"
                            ]
                        }
                    }
                }
            ]
        }
    ]
}

[asimfarooq5]

instance B looks like

{
    "$schema": "https://raw.githubusercontent.com/crossbario/crossbar/master/crossbar.json",
    "version": 2,
    "controller": {
    },
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "name": "realm1",
                    "roles": [
                        {
                            "name": "anonymous",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": false,
                                        "publisher": false
                                    },
                                    "cache": true
                                }
                            ]
                        },
                        {
                            "name": "router2router",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": true,
                                        "publisher": true
                                    },
                                    "cache": true
                                }
                            ]
                        }
                    ]

                }
            ],
            "transports": [
                {
                    "type": "rawsocket",
                    "endpoint": {
                        "type": "tcp",
                        "port": 6688,
                        "backlog": 1024
                    },
                    "options": {
                        "max_message_size": 1048576
                    },
                    "serializers": [
                        "cbor",
                        "msgpack",
                        "json"
                    ],
                    "auth": {
                        "cryptosign": {
                            "type": "static",
                            "principals": {
                                "edge": {
                                    "realm": "realm1",
                                    "role": "router2router",
                                    "authorized_keys": [
                                        "a476d81f9cd332638c3cb9cd927c6f6cc15a993eae3571e0c895ed5db42d9dc7"
                                    ]
                                }
                            }
                        }
                    }
                },
                {
                    "type": "web",
                    "endpoint": {
                        "type": "tcp",
                        "port": 8081,
                        "backlog": 1024
                    },
                    "paths": {
                        "ws": {
                            "type": "websocket",
                            "serializers": [
                                "cbor", "msgpack", "json"
                            ]
                        }
                    }
                }
            ]
        }
    ]
}

[om26er]

Thanks, I can reproduce that issue. Will work on a fix soonish

[om26er]

For this specific case the problem is that the instance A config's disclose stanza has "caller": false, which causes the following check to fail https://github.com/crossbario/crossbar/blob/b9799d2ac6667a97373db0549d2055fff97d784c/crossbar/worker/rlink.py#L257

In such instances, I believe crossbar should be a bit more helpful in logs

[om26er]

For this specific case the problem is that the instance A config's disclose stanza has "caller": false, which causes the following check to fail

https://github.com/crossbario/crossbar/blob/b9799d2ac6667a97373db0549d2055fff97d784c/crossbar/worker/rlink.py#L257

In such instances, I believe crossbar should be a bit more helpful in logs

sorry wrong, comment. That is a sub-bug that I discovered while using the configurations you provided. Anyways, the original issue still remains

[oberstet]

If the procedures are registered on instance A before the connection to instance B is established, it seems the invocations are not forwarded to instance B.

We have not yet covered all corner cases, eg startup order sequence like the one you ran into, with unit tests, and I can imagine there are bugs/gaps:(

The "normal" startup order is: 1) master node 2) edge nodes 3) all rlinks 4) start router cluster via master node

We should add integration tests for scenarios where order is different or dynamic.

---

fwiw, some background notes ...

There are multiple pieces that need to work together here.

What @om26er refered (worker/rlink) is the code in each managed node that implements the router-2-router links to other managed nodes. The code that should fire when a new procedure is registered on the far-node end of an rlink is also there. This is one piece.

Then there is the remote management bridge https://github.com/crossbario/crossbar/blob/b9799d2ac6667a97373db0549d2055fff97d784c/crossbar/edge/node/management.py from managed nodes to the master node.

Then there is code in https://github.com/crossbario/crossbar/blob/b9799d2ac6667a97373db0549d2055fff97d784c/crossbar/router/router.py (and dealer/broker) which use above remoting bridge eg to publish WAMP meta events when new procedures are registered (on a managed node router worker)

Such events originating from managed nodes are then processed in the master node

https://github.com/crossbario/crossbar/blob/b9799d2ac6667a97373db0549d2055fff97d784c/crossbar/master/arealm/arealm.py

[asimfarooq5]

Just verified, issue fixed in the linked PR.

[oberstet]

@asimfarooq5 thanks for testing and for reporting back! guess Omer is still polishing the PR, but will be merged then of course

[oberstet]

fixed via https://github.com/crossbario/crossbar/pull/1913