Ephemeral observer middleware

Description

The NATS JetStream event consumer have to distinct modes of operation for subscribers (consumers in NATS nomenclature); durable and non-durable. The difference is self descriptive but the usage of each has its own implications, and a sufficiently advanced implementation of eventhorizon will have to use both types.

Durable subscribers are retained indefinitely, with their position in the stream retained until manually removed. This is useful for eg. sagas that should process all events regardless of when they were produced.

Non-durable subscribers are removed a short time after client connection to NATS is lost, at the server's discretion. This is desirable when the subscription is short lived such as when a observer.RandomGroup or observer.HostnameGroup is used.

Other notes

Other event pub/sub systems might use other methods of removing stale subscriptions, such as periodic sweeping of old subscriptions. No such system exists for NATS JetStream AFAiK.

Affected Components

Event Bus, NATS JetStream, Middlewares

Related Issues

N/A

Solution and Design

Chosen implementation

An ephemeral middleware that can be used in conjunction with ie. observer groups in order to control the behaviour of event handlers and ultimately the behaviour of the event publisher.

When an ephemeral middleware has been identified and successfully queried for positive ephemeral status, the JetStream subscription is captured by an unsubscribe method and stored for later use. On graceful shutdown, the subscription is unsubscribed, thereby allowing NATS to remove the consumer and discard its position in the stream.

Other implementations considered

Non-durable subscribers are identified by a short random alpha-numeric ID, and as such are hard to identify and debug while in use. While non-durable subscriptions are desirable from a technical standpoint, there are benefits of using a subscribe/unsubscribe pattern with durable subscribers, such as:

Easily identifying both while running
Identifying the subscription of a crashed service that can be used for post-mortem analysis.

There is the possibility to communicate the ephemeral:ness of the handler via context, however this can feel like to much of a one-off and perhaps NATS specific. The ability to make a handler ephemeral is broader as most event publishers should have the need for some kind of clean up operation of short lived subscriptions. Implementing a configuration context for each type of event publisher even though it's the same property they configure is undesirable to me.

Even though, it could look like this:

ctx := natsEventBus.WithEphemeralHandlerContext(context.Background())

Steps to test and verify

Use the ephemeral middleware in conjunction with NATS JetStream as event bus:
```
if err := eventBus.AddHandler(context.Background(),
eh.MatchAll{},
eh.UseEventHandlerMiddleware(h,
    ephemeral.NewMiddleware(),
    observer.NewMiddleware(observer.RandomGroup()),
),
); err != nil {
...
}
```
This will capture all events for the duration of the service lifetime, at which point the subscription is unsubscribed and NATS can discard it.

looplab / eventhorizon