The following discusses a number of aspects and describes a design for an experimental prototype implementation of router-based resource / bandwidth limit controls for WAMP.

The goal of resource limit controls is to divide limited WAMP routing resources in a multi-user/role/realm in a user defined and controlled way.

Routing Resources

The computing resources consumed by Crossbar.io for core WAMP routing is primarily CPU, memory and network scaled by the following factors:

Static

number of connected TCP sockets (client connections accepted on the listening ports)
number of authenticated WAMP sessions joined on WAMP realms
number of WAMP realms
number of registered WAMP procedure endpoints in attached sessions
number of subscribed WAMP topic handlers in attached sessions

Dynamic

number of new WAMP client connections per second (including TCP, TLS and WebSocket opening handshakes)
number of new WAMP session authentications per second (everything between HELLO and WELCOME messages)
number of transport heartbeat / keep-alive messages per second on WAMP client transport connections

Dynamic WAMP Control

number of new WAMP procedure registrations (and unregistrations) per second from WAMP client sessions on realms
number of new WAMP topic subscriptions (and unsubscriptions) per second from WAMP client sessions on realms

Dynamic WAMP App Data

number of and application payload volume in WAMP calls and WAMP call results (including errors and progressive results) per second
number of and application payload volume in WAMP publications and events per second

Resource Limits

Scope

Resource limits can be applied at different resource scopes:

realm: resource limits apply to the sum/set of all sessions on the given realm
realm-authrole: resource limits apply to the sum/set of all sessions of the given authrole on the respective realm
realm-authrole-authid: resource limits apply to the sum/set of all sessions of the given authid under the respective authrole and realm

Dimensions

Resource limits can be applied along different resource dimensions:

number of sessions
number of registrations
number of subscriptions
bandwidth in message traffic volume per second
- guaranteed (per-session in kbps) - this is promised to be available continously and long-term
- burst (per-session in kbps) - this may or may not be available for short-term bursts (needs to recover to be available again)
- total (in kbps over all sessions in a scope) - hard limit beyond which resources will be denied or actively killed

A resource limits configuration stanza may therefor look like:

{
    "sessions": 100,
    "subscriptions": 500,
    "registrations": 500,
    "bandwidth": {
        "guaranteed": 200,
        "burst": 200,
        "total": 1000
    }
}

Here, "bandwidth" is following this rational / is to be understood in this way: "bandwidth" is based on sum of the uplink and downlink volume Bytes in terms of serialized WAMP messages at the WAMP transport level (e.g. before WebSocket or RawSocket framing and TLS encryption, but after CBOR or other serialization)

FIXME: Discuss multi-node cluster setups where the scope may apply at a per worker, per node or per cluster level.

Node Configuration

The following is an example of a (incomplete) node configuration that illustrates the application of resource limits at different scopes (realm, role and authid), and using only one resource control dimension "sessions":

{
    "version": 2,
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "name": "realm1",
                    "roles": [
                        {
                            "name": "anonymous",
                            "permissions": [],
                            "limits": {
                                "sessions": 1200
                            }
                        },
                        {
                            "name": "frontend",
                            "permissions": [],
                            "limits": {
                                "sessions": 800
                            }
                        }
                    ],
                    "limits": {
                        "sessions": 2000
                    }
                }
            ],
            "transports": [
                {
                    "type": "websocket",
                    "endpoint": {
                       "type": "tcp",
                       "port": 8080
                    },
                    "auth": {
                        "cookie": {},
                        "anonymous": {
                            "type": "static",
                            "realm": "realm1",
                            "role": "anonymous",
                            "authid": "${CBTID}",
                            "limits": {
                                "sessions": 10
                            }
                        },
                        "wampcra": {
                            "type": "static",
                            "users": {
                                "alice": {
                                    "secret": "${ALICE_SECRET}",
                                    "realm": "realm1",
                                    "role": "frontend",
                                    "limits": {
                                        "sessions": 600
                                    }
                                },
                                "bob": {
                                    "secret": "${BOB_SECRET}",
                                    "realm": "realm1",
                                    "role": "frontend",
                                    "limits": {
                                        "sessions": 400
                                    }
                                }
                            }
                        }
                    }
                }
            ]
        }
    ]
}

Above configuration would lead to the following behavior:

Alice may connect up to 600 (authenticated) sessions to realm1
...

Resource Management

Resource Policing

FIXME: discuss "resource policing" vs "resource shaping" as it applies for WAMP (background here and here)

Meta API

wamp.limit.authid.on_limit_exceeded
wamp.limit.authrole.on_limit_exceeded
wamp.limit.realm.on_limit_exceeded

with arguments

authid|str or authrole|str or realm|str
period_no|int
period_start|int
period_end|int
limit|int
actual|int
is_warning|bool

and (in case of warnings)

wamp.limit.authid.on_limit_recovered
wamp.limit.authrole.on_limit_recovered
wamp.limit.realm.on_limit_recovered

One WAMP meta procedure registered:

wamp.reservation.get(delegate_adr) -> reservation

One user WAMP procedure called:

wamp.delegate.<delegate.address>.claim_reservation(reservation.address, ...)

Policy Enforcement

wamp.session.kill(session_id|int, reason|str, message|str)
wamp.session.kill_by_authid(authid|str, reason|str, message|str)
wamp.session.kill_by_authrole(authrole|str, reason|str, message|str)

Implementation

IPendingAuth.hello https://github.com/crossbario/crossbar/blob/09eaf37ce0dfc58e97ed8e0af428bea28dea61cc/crossbar/interfaces.py#L72
Accept.limits https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/types.py#L134
Welcome.limits https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/message.py#L840
Goodbye.reason == "wamp.close.limit-exceeded" https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/message.py#L1327
CloseDetails.reason == "wamp.close.limit-exceeded" https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/types.py#L422

Dynamic Authenticators

Consider

{
    "pubkey": "...",
    "realm": "realm1",
    "authid": "alice",
    "role": "frontend",
    "extra": {"message": "Welcome, Alice!"},
    "cache": true,
    "limits": {
        "sessions": 400
    }
}

here

https://github.com/crossbario/crossbar-examples/blob/852680eee646cf5479bff18ec727a8026d9bdcda/authentication/cryptosign/dynamic/authenticator.py#L70

This would apply the authid-level ("alice") resource limits, that is "at most 400 sessions of alice" are accepted.

One question that needs to be answered when coding is how to combine the authid-level limits with any limits defined at the authrole and realm level.

Resource management database tables

add to https://github.com/crossbario/cfxdb/blob/master/cfxdb/realmstore.fbs
period_no|int is the numbered time period, 10-20s long

cfxdb.realmstore.ResourceLimit

limit_oid|uuid
realm|str
authrole|str
authid|str
sessions|int
subscriptions|int
registrations|int
bandwidth_guaranteed|int
bandwidth_burst|int
bandwidth_total|int

cfxdb.realmstore.ResourceReservation

reservation_oid|uuid

cfxdb.realmstore.ResourceReservationClaim

claim_oid|uuid
reservation_oid|uuid

cfxdb.realmstore.ResourceUsage

period_no|int
limit_oid|uuid
reservation_oid|uuid
last_claim_oid|uuid
node_oid|uuid
node_authid|str
worker_name|str
worker_pid|int
realm|str
authrole|str
authid|str
sessions|int
subscriptions|int
registrations|int
uplink_volume|int
downlink_volume|int
limit_warnings|int
limit_enforcements|int

crossbario / crossbar

Resource limit controls and meta API #2017