jet / propulsion

.NET event stream projection and scheduling platform with CosmosDB, DynamoDB, EventStoreDB, MemoryStore, message-db, Equinox and Kafka integrations
https://github.com/jet/dotnet-templates
Apache License 2.0
178 stars 24 forks source link

feat(Cosmos): Implement Monitor/RunUntilCaughtUp #235

Closed bartelink closed 10 months ago

bartelink commented 1 year ago

In order to be able to wait for reactions to complete In order to be able to stop runs of tools when the associated checkpoint has caught up As a dev, I want to be able to do a RunUntilCaughtUp and/or Monitor.AwaitCompletion as I can for the other stores


Cosmos doesn't yet implement a Monitor for various reasons:

  1. there's no obvious way to infer from a given batch of change feed docs whether they represent the tail
  2. the Change Feed Lag estimator runs async to the read process
  3. the Change Feed Lag estimator is reliant on the position update having been persisted
  4. there is no active reader process that we control i.e. the reader/observer callback is not triggered if there are no documents to process

Ideas

One consideration is there's different SLA expectations depending on the use case, e.g.

bartelink commented 10 months ago

Implemented in #238 - Cosmos now has a parallel impl of the Feed structures, and the FeedMonitor and interfaces live in the main Propulsion module

This opens up the chance to unify Propulsion.Kafka too. The Ingester/Submitter pieces and checkpointing responsibilities are currently 'where they landed'; this ad hoc structure can, with some thought, likely be simplified.

The chances are the changes will be relatively minor in terms of surface API impact; the payoff will be in being able to e.g. look at the Kafka impl (which can checkpoint via a synchronous write to an offset property), and contrast it with Cosmos and/or the Feed-based ones.