bluesky / event-model

data model for event-based data collection and analysis
https://blueskyproject.io/event-model
BSD 3-Clause "New" or "Revised" License
15 stars 31 forks source link

Add support for chaining DocumentRouters. #178

Closed danielballan closed 4 years ago

danielballan commented 4 years ago

This addresses a long-considered need in DocumentRouter: the ability to chain them.

Description

A common use case would be connecting a DocumentRouter that does some transformation (e.g. dark subtraction) to a DocumentRouter that does visualization or serialization. Currently, the only way to do this is:

def factory(name, doc):
    router3 = Router3(...)
    router2 = Router2(...)
    router1 = Router1(...)

    def cb(name, doc):
        name1, doc1 = router1(name, doc)
        name2, doc2 = router2(name1, doc1)
        router3(name2, doc2)
    return [cb], []

RunRouter([factory])

which is verbose and error-prone. This PR enables:

def factory(name, doc):
    router3 = Router3(...)
    router2 = Router2(..., emit=router3)
    router1 = Router1(..., emit=router2)
    return [router1], []

RunRouter([factory])

Design Considerations

  1. The relationship between documents received and documents emitted does not have to be 1:1. Note that the base class does not call emit internally at all. It is up to subclasses to define whether and when to emit. For example, Serializer classes and classes that produce visualizations are generally "sinks" and have no need to incur the overhead of emitting the documents to a downstream node. Some applications (example) may one to emit multiple documents when they receive a single document.
  2. This emits straightforwardly to at most one downstream node. For more complex behavior, like fanning out to multiple nodes, handling backpressure, and other considerations, the user should engage a more complex library like streamz by registering a Stream node with emit. By accepting this one additional layer of indirection, we keep these considerations separate from event-model.

Alternatives Considered

We could leave it up to subclasses to add emit but a standard emit method and __init__ parameter

danielballan commented 4 years ago

Good suggestions. Addressed.