Registration-based router processing ($100)

[MaddieM4]

Still working out in my head how this is going to work, but I'd like to have an official place to share and bounce ideas off everybody, rather than keep it cramped and musty in my own attic.

We want some kind of prioritized, registered chain of callbacks for handling messages in the router, which will open up the door for future extensions such as persistent stream support, applying onion routing to arbitrary frames, etc. The final steps will be the ones that are primary now - send to a client if available, and send to a jack if available.

Development timing

Probably best to wait until both jack and frame registration are implemented, but that doesn't mean we have to wait to cook up ideas.

Prioritizing callback execution

Each callback is registered with a name, and a function.

The name follows a format of "7 Route into Hyperboria", consisting of a number, a space, and a short descriptive title of what the function does. Prioritization occurs by string-sorting these keys. All the names starting with "1" will be checked before those that start with "2", and "404" will happen before "45". We can organically extend specificity as far as we want, and numbers can collide as long as the rest of the name doesn't (which is great for programmatically generated rerouting callbacks).

We should probably define some basic categories for the single-digit numbers. 9 should definitely be reserved for jack and client dispatch. The rest, I dunno. We certainly don't have to define them all yet.

Function signature and behavior

A callback function takes arguments (router, frame, resend). It should use a docstring to provide a longer description of its purpose and inner workings.

resend(frame) is a function that sends a frame through another pass of the callback list. Callbacks are encouraged to use this function, rather than router.recv(frame), because it internally imposes a recursion limit. It is defined on the fly at the top of router.recv.

There may be efficiency concerns to this approach, so I'm open to an integer-based alternative to the same problem.

A function signals if a message should be passed to the next callback by the return value. True if the callback has handled things and no further processing is necessary, False to let the next callback in the chain take a crack at it. If the callback raises an exception, the frame is passed to the next callback as if the function had returned False, and the traceback is logged as an error.

Registration

Largely depends how we implement registration elsewhere. I don't think we want global registration, since callbacks are basically the configuration of routers, and while most applications will only have use for one router, there are use cases for separate and differently configured routers within the same Python environment.

That does bring up the question of how to register default callbacks. I propose doing a call to a setup_default_callbacks or similar function in router.__init__, which registers a small set of basic callbacks, probably just jack/client dispatch. Leave it up to the application to build up from that minimal base as the application needs, rather than bloating the default configuration.

[MoritzS]

That will also make debugging easier. You will just add a debugger callback that prints the contents of each packet it receives or some other information. This can be used in applications that use EJTP for extended authentication, too, since they can check each Frame for its validity on their own. This approach reminds me of django's middleware system, which also allows the callbacks to add meta data to the requests (or in our case Frames).

[MaddieM4]

Callback order:

1. Optional logging/filtering
2. Routing transformations
3. Third-party middleware/plugins
4. RESERVED
5. RESERVED
6. RESERVED
7. Transport-level transformations (compression, for example)
8. Network/client dispatch
9. Error handling

Recursion limiting

Instead of a resend function, pass in a countdown generator created with range(self.recursion_limit). This provides a lightweight tracker that runs out after self.recursion_limit calls of baton.next().

class RouterRecursionCrash(Exception): pass

class ExampleRouter:

    def __init__(self):
        self.baton = None
        self.recursion_limit = 10

    def recv(frame):
        if not self.baton:
            # Top level of recursion - resets baton regardless of failure or success.
            self.baton = iter(range(self.recursion_limit))
            try:
                return self.recv(frame)
            finally:
                self.baton = None

        try:
            self.baton.next()
        except:
            logging.info("Hit recursion limit, dropped frame", frame)
            raise RouterRecursionCrash()

        for callback in sorted(self.callbacks.keys()):
            # Callbacks should pass baton as an
            # argument to their calls to router.recv
            try:
                result = callback(self, frame)
            except RouterRecursionCrash:
                raise RouterRecursionCrash() # Bubble this one error up
            except Exception as e:
                logging.error(e)
                result = None
            if result:
                break

Registration

Should be an instancemethod that acts as a decoration. We ought to check if the following is possible in python syntax:

class ExampleRouter:
    def registerCallback(self, title):
        return lambda func: self._callbacks[title] = func

    # Default callbacks

    @self.registerCallback('80 Dispatch to local clients')
    def clientDispatch(self, router, frame, baton):
        # Look for local client matching frame.receiver

    @self.registerCallback('82 Dispatch to local jacks')
    def jackDispatch(self, router, frame, baton):
        # Look for jack matching frame.receiver
        # Create writer jack if necessary

But at the very least, it gets us this:

myrouter = Router()

@myrouter.registerCallback('7 Zlib Compression on TCP traffic')
def compress_tcp(router, frame, baton):
    ...

[MaddieM4]

Bounty is $100.

http://www.freedomsponsors.org/core/issue/198/registration-based-router-processing

---

(Copied from acceptance criteria)

Create callback registration-based Router system, for on-the-fly extension and introspectability, based on design discussion in issue tracker.