Process Manager

[codingedgar]

What's the model to do Process Managers/Sagas/Long running processes or in general, transforming event -> command list?

I don't understand well the proposed framework, sorry 😅

Looks like everything is focused on the Command side (command -> 'state -> event' list) and queries are resolve based on Aggregate state, like "commands with result" (command -> 'state -> result * event' list), am I right?

I understand that propulsion is meant for subscriptions/projections, but then how is the Query part (of CQRS) fall into place within the Equinox Framework?

I'll give you an example to illustrate my confusion:

• C: Command
• E: Event

A User Service (like a server) sends a command to a Crawler Aggregate to start the crawling, but the HTTP requests need to go through HTTP Aggregate that will schedule each request and process them later (due rate limits).

When the HTTP Aggregate process the requests it issues an event with the response and a Process Manager "should" pick the events and issue commands to the Crawl Aggregate.

• User Service ->[C:Start Crawl Process] -> Crawler Aggregate -> (OK, [E:Crawl Process Started])
• E:Crawl Process Started -> Crawler Process Manager -> [C:Request HTTP] -> HTTP Aggregate -> (OK, [E:Request Scheduled])
• HTTP Aggregate -> [E:Request processed]
• E:Request processed -> HTTP Process Manager -> [C:Response delivery] -> Crawler Aggregate -> (OK, [E: Page analysed; E:Crawl Process finished])

I understand the aggregates (command -> 'state -> event' list) validate the domain and produce events and I'm using the Process Managers(event -> command' list) as asyn communication mechanism between Aggregates.

The parts I don't understand is how the Crawler Process Manager & HTTP Process Manager should be implemented using Equinox, or if that's not the right model for Equinox, then what would be a valid model to achieve this example?

[bartelink]

Firstly, thanks for the interest and taking the time to lay out your question so clearly.

Yes, the query side side, taking a CQRS model is intended to be managed by building projections via Propulsion.

There are no plans for any specific process management extensions in Equinox or Propulsion as such; in my view, the Equinox primitives, together with F#'s affordances for managing state machines (expressing folds as DUs) are sufficient to let one succinctly write what's required in an Explicit is better than Implicit fashion (i.e. instead of frameworking it - providing something that forces a way to write a Process Manager, including making you have to hook into some set way of doing things by establishing concepts that one must express things in terms of). That's not to say small helpers can't be added to make things easier in common cases

https://github.com/jet/dotnet-templates/pull/40 shows a process manager (https://github.com/jet/dotnet-templates/pull/50.is a better entry point as it simplifies some aspects).

On my (medium term, no commitments, virtual) roadmap, that example/template also gains some diagrams and a bit of a writeup regarding the general patterns involved - whether that goes into https://github.com/jet/dotnet-templates#patterns--guidance, the readme of https://github.com/jet/dotnet-templates/pulls/50, or into the DOCUMENTATION.md in here is an open question - certainly any contributions, either in those locations, or elsewhere externally would be more than welcome.

I hope this begins to answer your need; please feel free to ask follow-ups, I know this is all very handwavy!

[codingedgar]

Thank you for responding so quickly @bartelink 🎉

I'm new to F# and .NET, though I have some knowledge of C# and have been coding for 7 years, I might ask questions and get in trouble with the basics of F# or .NET because I'm still reading basic books about F#, sorry.

This examples (both equinox/samples & jet/dotnet-samples) are however kind of difficult to understand for me, as a newcomer (to equinox), it looks like support material for a presentation given by someone who already understands them.

To understand the problem a example is solving I would read the README file and if I cannot read the "problem & proposed solution with diagrams" (like a child i know 👦) I would go straight to the tests (integration tests or BDD style) because they have all the spec. So i'm missing the spec of this examples, I think that the Todo Backend is the only problem I kind of understand the reference of. I'm sure they are great examples I just don't know how to read them, they are filled with technical details, but I can't get past the "what is this doing?" part, followed by the "but why like this? 🤔"

From:

STORE, see /samples/Store

The core sample in this repo is the Store sample, which contains code and tests extracted from real implementations (with minor simplifications in some cases).

These facts mean that:

• some of the code may be less than approachable for a beginner (e.g. some of the code is in its present form for reasons of efficiency)
• some of the code may not represent official best practice guidance that the authors would necessarily stand over (e.g., the CQRS pattern is not strictly adhered to in all circumstances; some command designs are not completely correct from an idempotency perspective)

While these things can of course be perfected through PRs, this is definitely not top of the TODO list for the purposes of this repo. (We'd be delighted to place links to other samples, including cleanups / rewrites of these samples written with different testing platforms, web platforms, or DDD/CQRS/ES design flavors right here).

I pick up on:

• the code may be less than approachable for a beginner

• this is definitely not top of the TODO list for the purposes of this repo

• We'd be delighted to place links to other samples

Based on this, I think I get the spirit of the project, and seems great, I'll do some tests and write some examples and will try to PR them here to check if I understand correctly the proposal to Equinox and Propulsion.

If there's anything regarding "understanding" and "explaining" this projects that you think I could help (event it is making a napkin draw into a diagram) let me know 👍 .

Regarding jet/dotnet-templates#40 specifically the Process Manager looks like it's exactly what i'm looking for. The system I'm building has agents to automate tasks, based on time or reacting events. However I'm not sure if I'm reading right or missing something but this Process Manager is actually a stream (event -> event' list) meaning it reads from one event stream and writes to another event stream.

I'm looking for Process Manager with form of event -> command' list because that command goes to an Aggregate that keeps everything sound (as the solely writer of one event stream), I'm yet to familiarize with Streams (Process), I still don't wrap my head around on how to prevent multiple writers to one event stream. In a Many-to-one scenario with simple Process Managers, many can talk to one Aggregate and that one is the owner of its event stream, but with many Streams then each writes the same output event stream? Or One Stream has to merge input event streams in order to be the only writer to one event stream, but then where to decouple? Sorry for my ignorance with Streams Processes 😅 , I know that those questions have simple answers that I'm yet to discover, but I'm sticking to basic Aggregates/Process Managers until I feel comfortable enough to take the next step (Streams Processes).

[bartelink]

Really appreciate the setting context

This examples (both equinox/samples & jet/dotnet-samples) are however kind of difficult to understand for me, as a newcomer (to equinox), it looks like support material for a presentation given by someone who already understands them.

Yep; its a tough tradeoff - simple, high performance, testable, easy to scan. Diagrams need to happen and will - they just compete with lots of things...

There unfortunately are no tests I can point to (and FsCheck ones are the worst possible start unless that's something you happen to already grok.)

Based on this, I think I get the spirit of the project, and seems great, I'll do some tests and write some examples and will try to PR them here to check if I understand correctly the proposal to Equinox and Propulsion.

Looking forward to it; I'll endeavour to stay responsive.

Regarding jet/dotnet-templates#40 specifically the Process Manager looks like it's exactly what i'm looking for. The system I'm building has agents to automate tasks, based on time or reacting events. However I'm not sure if I'm reading right or missing something but this Process Manager is actually a stream (event -> event' list) meaning it reads from one event stream and writes to another event stream.

Approximately; however I'd say it's 'event ->'action`; you can map the state of the process manager to a "next step with gathered context" as you see fit. How you carry out those actions is you writing code, i.e. while this might appear to just write an event, it's running a command, whose outcome will dictate what you feed back into the process manager (which in turn influences the next action)

I'm looking for Process Manager with form of event -> command' list because that command goes to an Aggregate that keeps everything sound (as the solely writer of one event stream), I'm yet to familiarize with Streams (Process), I still don't wrap my head around on how to prevent multiple writers to one event stream.

I'm having trouble unpacking this. Also you don't "prevent" multiple writers, you define what happens if they happen to conflict (before that you do want to define aggregates as much as possible to minimize such conflicts)

In a Many-to-one scenario with simple Process Managers, many can talk to one Aggregate and that one is the owner of its event stream, but with many Streams then each writes the same output event stream? Or One Stream has to merge input event streams in order to be the only writer to one event stream, but then where to decouple? Sorry for my ignorance with Streams Processes 😅 , I know that those questions have simple answers that I'm yet to discover, but I'm sticking to basic Aggregates/Process Managers until I feel comfortable enough to take the next step (Streams Processes).

On reflection, while its technically possible to model a web crawler as a process manager, it's not feeling like there's a fit to warrant applying event sourcing:

• there's nothing to collaborate on - I'm not seeing/feeling anything that needs to be consistent, agreed and/or have a log of
• there's no strict ordering constraint; you're just adding stuff to a queue (not ordered, it could be a bag, could use CRDTs)
• there's no end to the life of the process, no intrinsic batching
• if we do some double visits at statistically low levels, it's not the end of the world etc.

I can't say I've an alternate approach to point to off the top of my head (I prefer to do work than on real things than expand my system design interview skills 😸), but I'm sure there's plenty well documented approaches to managing such an activity a search away - bloom filters, CRDTs, consistent hashing etc. to distribute the work across multiple nodes tc

[codingedgar]

How you carry out those actions is you writing cod

Oh I see now! Of course, I can change what the Process Manager does. I see the event -> * form now.

it's not feeling like there's a fit to warrant applying event sourcing

Oh the crawler and http services are just one tip of the system, I see it might not seem to need ES (and it might not, too noob to know, I’m just sure that I need more that the present state) but I need to analyze the responses and behavior of both services as an ordered stream of events to achieve other parts of my project (for example performance tuning, stats growth over time, caching & invalidations), complying with all the 3 bullet points. I will make some videos explaining this in YouTube, I’ll point to relevant ones in following questions to make it easy to explain stuff.

Thank you so much for your answers! 💛

[bartelink]

One thing to watch out for is that you do want to arrive at a situation where each stream has a natural end of life in some fashion - talk. The Epoch mechanism in the process manager PRs is a way to synthetically introduce such a thing, but be careful trying to use that as a hammer; thinking deeply about a good way to structure things is not optional - talk.

Finally, there's your use of complying in reference to my approx reasons why event sourcing is inappropriate for a context - be careful of confirmation bias in justifying a need (but that doesn't mean you need to go the other way and take official-sounding classification rules literally either).

None of this is a reason not to just build something in order to explore a range of impl techniques. If the intention is to build some hypothetical domain with a view to demonstrating some technical techniques, go for it (but the whole crawler should really not be a single process manager, no matter what!)

[codingedgar]

WOW ! thank you for that knowledge 😻 !

I'll watch the talks!

That article is awesome! I've been thinking long and hard about what to do, and still think is the right approach, but might be just my ignorance ATM, I'll keep an eye to know if i'm not making a mistake, that's exactly why I want to explain this project of mine on YouTube, to have people take a look at it and point out mistakes, I'm a single dev trying to build a product for a real comercial need.

The whole crawler is not a single process manager, nor is just one crawler, nor is just one app that solely crawls 🦀 🕷 , that's just one of multiple data ingestion mechanisms, the target page (actually is multiple APIs and web pages) have very complex limitations to data collection and therefore the complexity and decoupling between the crawler (how to crawl) and the actual http request (when to request). I'll comeback with a video explanation of this in the coming weeks, to further illustrate my conclusions on why this arrangement.

The crawler has natural end, and it crawls based on each user need, but it has cycles. I was thinking on putting everything on the same crawling-on-behalf-of-user-{used_id}-stream and make no distinction of this cycles, but I'll watch the talks and get to understand why I need to do crawling-on-behalf-of-user-{used_id}-{cycle_id}-stream, or if I'm just understanding this wrong, this is related to the comment i left here

Representing a long-running state in this fashion is no panacea

In what fashion?

in modeling a system, the ideal is to have streams that have a naturally constrained number of events over their lifetime.

I'm a newcomer to Event Sourcing, is there any reference (book) pointing to this part?

Then if a stream that is long lived like "user actions in the UI", it would be more ideal to partition this for example in "sessions" like having user-actions-in-the-UI-{user_id}-{session_id} streams ? because sessions are finite, or that would be bad modeled as well, not sure if i'm missing the point.

That arise more questions for me like "stream explosion" (referencing state-explosion) or "stream discovery", but stream discovery is less of an issue with categories and stream indexes/queries/projections.

I'll take the fashion is "Epoch mechanism synthetically introduce constraided number of events", and that the reference are those talks.

Thank you very much for your time and insight really 🎯 💛

[bartelink]

My ref to Epoch in the above refers to how https://github.com/jet/dotnet-templates/pull/40 synthetically introduces epochs into the location streams and the inventory stream in order to reduce the amount of events in a given scope (maybe Tranche speaks to you more as a term for that).

Sadly, there's no book, but it's inevitable there will be one. The closest concrete thing I know to having such a thing is lurking in the general and eventsourcing channels in https://github.com/ddd-cqrs-es/slack-community.

Not sure user actions in the UI would be a stream - i.e. you may be dumping stuff into logical or physical queues, but aggregates exists as logical grouping of things for a reason - there needs to be some invariant being maintained etc.