search

jet / dotnet-templates

Example app and service templates `dotnet new -i Equinox.Templates; dotnet new eqx*/pro*` https://github.com/jet/equinox https://github.com/jet/FsCodec
https://github.com/jet/propulsion
Apache License 2.0
64 stars 16 forks source link

Is `proReactor`'s Ingester actually a Reactor? #92

Closed dharmaturtle closed 2 years ago

dharmaturtle commented 3 years ago

Link to file in question.

I ask because the Propulsion docs state

Ingesters ... are not acting in reaction to events emanating from the Consistent Event Store

It further states:

Reactors: drive reactive actions triggered by either upstream feeds, or events observed in the Consistent Event Store

My reading of ReactorTemplate.Ingester.handle is that it is acting in reaction to events from the event store. In particular, it is passing events to TryIngest, which apparently tries to Consume the event. Am I correct in thinking that Ingester is a Reactor, or are there nuances/grey areas here that I'm missing? Things might be complicated because it is producing its own event stream for reasons I currently don't grok. (The events are then treated as Commands, which breaks my brain.) I might be being too pedantic and you mean this as a more throwaway example.

Ultimately, I'm trying to figure out what to base my (non Kafka) projector code on. The proReactor template looks promising, and instead of a TodoSummary module I'll have a TodoProjector which writes to Azure Table Storage/ElasticSearch in response to events.

bartelink commented 3 years ago

Hm, a good question!

The intent in the docs is to call out specific families of 'projectors' and 'services' in a system in order to allow a higher level vocabulary to be applied. In that context, an Ingester is a specific kind of reactor whose primary job is to take in data and stash it in a suitable way for this overall bounded context.

Where an ingester writes into an event sourced aggregate (but, even if you run a CFP with RollingState - you'll still see each change guarantee a handler invocation), it actually is doing more than the strict act of 'take it and stash it'.

The name of the file Ingester.fs in the reactor template results from a combination of: b) being copied from summaryConsumer (that template is definitely an Ingester overall) a) from the fact that proReactor is about 6 templates in one (which is why program.fs is a nightmare of ifdefs) - there is already a Handler.fs which is used for the 'real reactors' and for some that are more like publishers.

My attempt to define ingesters vs publishers vs reactors happened after this... After https://github.com/jet/dotnet-templates/pull/91 is merged, it might be a good idea to rename Ingester.fs in all the contexts to just be Handler.fs where possible (but that's still hard in proReactor).

The events are then treated as Commands, which breaks my brain.

That's what a reactor should be doing by definition. However when ingesting/denormalizing things from an event source, you generally want to do that processing idempotently in order to ensure that rewinding the offsets on the source does not e.g. destroy or add conflicting information - e.g. if your read model has a projection from V5 of a stream, and e.g. the projector is restarted before it can save its offsets and/or you reset them and/or you are working from a source that can present events multiple times (hint: that's really all sources - At Least Once Delivery is a reality for most things and thats what Propulsion centers on and most logic in most Handlers should be considering that).

Regarding your actual need, proProjector should cover your need in this instance, but proReactor -blank would work fine too. To be honest, if I had time and was focusing on the understandability of the templates, my next tasks would be:

  1. remove Ingester.fs names
  2. make a diagram like the overview one in Propulsion that maps it to templates in dotnet-templates and add that to templates's overview.md
  3. look at proReactor and proProjector together and factor out the needs into 2-4 templates covering a. example wiring for each source b. examples of publishing to kafka c. blank vs having something that reacts to events by stashing/denormalizing to a read model (aka a projector/denormalizer) vs a reactor that does event sourced things / runs commands/makes decisions

I'd take and/or be happy to work with you on a PR that kicks off that journey by e.g. taking proProjector, taking out lots of stuff (esp Kafka - proProjector and pushing to kafka was the first use case but in the greater scheme of things it really should not be that central). I guess that would be a proDenormalizer /proReadmodelUpdater or hopefully a better name !

I think this is the best place for us to have this discussion but feel free to reach out on the DDD-CQRS-ES slack's equinox channel too...

bartelink commented 2 years ago

@dharmaturtle we never really put this to bed properly

Since the above, thanks to @ragiano215's initiative (and the nudge you provided by asking the question and getting me to put out my thoughts above) we have a proReactorCosmos template which is much more legible. It calls the file with the handler within it Reactor.fs and the Reaction event filtering has a specific section and patterns now: https://github.com/jet/dotnet-templates/blob/master/propulsion-cosmos-reactor/Todo.fs#L23-L31

I have intentions to add a Reactor to https://github.com/oskardudycz/EventSourcing.NetCore/pull/84 which will expose a Feed Source based on the pattern in https://github.com/jet/dotnet-templates/tree/master/equinox-patterns#list-epochsseries-with-exactly-once-ingestion-guarantee

At this point I'm going to close this issue; feel free to ask any follow-ups and/or call me out on stuff that's unclear/dubious