Feature request, commit multiple event streams in one go

leidegre commented 8 years ago

I have a situation where I have dependencies between aggregates parent/child relationship. In this case, it would be really useful if it would be possible to say that I wanted to save changes of two (or more) event streams as a single transaction.

Similar to how multiple events are stored using an intermediate transaction document.

If there's a better way to do this, I'm all ears.

adrai commented 8 years ago

in a distributed system... this is not really possible... so I think this "feature" will not land here... :-(

leidegre commented 8 years ago

@adrai OK, yeah that's definitely a hard problem but not impossible. Just look at what Google is doing with Spanner. Not saying, we should compete with Google ;)

I spent some time reading the source code yesterday (I was looking at the mongodb provider specifically). It already does implement a sort of transaction handling. Would it really be so bad to have something that allowed me to commit multiple streams in a similar manner? It wouldn't have to be atomic if the provider couldn't support it a sensible manner but it would be nice if you could express that intent that you wanted to persist a set of events as a group transaction or sum such.

Thoughts?

PS. I'll totally take charge on this with your blessing unless you feel strongly against this.

adrai commented 8 years ago

Ok... try to do a PR... :-)

leidegre commented 8 years ago

@adrai I'll try to get a working prototype with mongodb (since that's what I'm working with right now) done and we can talk about what we'll do from there. No commitments just exploring the possibilities. To be continued.

adrai commented 8 years ago

ok

mpseidel commented 8 years ago

@leidegre would you mind sharing what kind of aggregates you are working with? What's the domain you are working in and what are you trying to achieve?

I am asking because usually your aggregates should be your consistency boundary and transactions across aggregates should not be necessary as I understand the concept.

mpseidel commented 8 years ago

In the case of a parent/child relationship - shouldn't all command be issued against the parent (root) aggregate and so also all events being raised come from that root aggregate?

adrai commented 8 years ago

@mpseidel +1 :-)

but what if somebody do use an eventstore without implementing cqrs?

mpseidel commented 8 years ago

@adrai well - if you are not trying to implement it the ddd/cqrs way then it might be a totally different situation and the use case could be valuebale. but the implementation as of now is written to explicetly use constructs of ddd as we have aggregate and aggregateId in the codebase for example.

implementing and supporting it could lead to problems though especially with supporting multiple data stores. azure tables for example will not be able to handle it unless you implement it so that you dont use the aggregateId as a PartitionKey (I actually had to do that myself just to support multitenancy).

It depends I guess :)

adrai commented 8 years ago

You're right!

Il giorno 9-mar-2016, alle ore 07:25, Michael Seidel notifications@github.com<mailto:notifications@github.com> ha scritto:

@adraihttps://github.com/adrai well - if you are not trying to implement it the ddd/cqrs way then it might be a totally different situation and the use case could be valuebale. but the implementation as of now is written to explicetly use constructs of ddd as we have aggregate and aggregateId in the codebase for example.

implementing and supporting it could lead to problems though especially with supporting multiple data stores. azure tables for example will not be able to handle it unless you implement it so that you dont use the aggregateId as a PartitionKey (I actually had to do that myself just to support multitenancy).

It depends I guess :)

Reply to this email directly or view it on GitHubhttps://github.com/adrai/node-eventstore/issues/63#issuecomment-194138699.

leidegre commented 8 years ago

@mpseidel It's like this,

We have hotels and a hotel can have a single hotel as its parent. Further, a hotel that is the parent of some hotels also have these hotels readily available as children.

You can have hotel A the parent of hotel B and C but hotel B and C cannot have any other parent (or be child of any other hotel).

The way our infrastructure works is that we have an ID and aggregate type. We then construct an object that represent that aggregate based on its event stream. The object supports logic and validation for making these parent/child relationships and we pass the objects as arguments to the methods that create the new relationships.

Hotel A adds Hotel B as child
Hotel B sets Hotel A as parent

These are two independent operations on two different aggregates but we would very much like to commit the changes as a whole. This way, a validation error in either operation would cause the transaction to fail.

I don't know what other possible structure would be able to create the same result without having the same complexities but I'd very much like to know if there's alternatives to the strategies outlined here.

mpseidel commented 8 years ago

@leidegre thanks for elaborating. My girlfriend does know more about the hotel industry than me but I am assuming that you are tyring to model hotelchains and hotels that are belonging to that chain or is it something completely different? What makes a hotel A the parent of hotel B in the real world?

Anyways: Following DDD practices I would first do some context mapping and decide in what context I would implement the hotel parent-child-relationship concept in. Then I would probably need to know:

in what real world business context does the parent-child relationship matter? reporting maybe?
Is there an existing context in that you already use hotel aggregates? Do you need the information there or is it just some additional information displayed somewhere?
What would be the business impact or risk of expensive errors if Hotal A became parent of Hotel B but Hotel B executes some operation before it gets the information that it has a new parent?

One alternative would be to model the hotel-structure as such within one single aggregate "Hotel-Structure" that describes the relationship state of all hotels and their children so there is no possibility of consistency problems when defining the relationships. Then you could subscribe to events an use the structure information in other aggregates or even other contexts to use the information about the structural changes.

If you don't mind sharing more info it would help to understand your needs better - I'd love to learn more about the problem - I struggle with those kind of decisions constantly and the only thing that helps is experience from solving similar problems :)

leidegre commented 8 years ago

@mpseidel thanks, this is great, I'll try to be more descriptive. This is a great learning experience for me.

What we're trying to do is to create a group of hotel buildings. A standalone hotel is a hotel that has no parent (does not belong to a hotel complex (or grouping)). A hotel complex is a hotel that has one or more hotel buildings (children) and a hotel building is simply a hotel with a parent. Note the strict parent-child relationship and the fact that we don't go beyond 1 level, so no nesting. And you may not say that a hotel building belongs to more than 1 complex (zero or one).

In the business we don't differentiate between hotel complexes and standalone hotels. All we care about is hotels but in reporting situations there are those who want to know the numbers for a group of hotels.

One thing that came up was that we didn't want to invent the concept of a group and then attach stuff to that. There should just be hotels (this might seem a bit arbitrary and maybe it is but it was the primary reason we went with this design). Also, we're trying to minimize the risk of inconsistency even if everything is in truth eventually consistent.

I'm a data person. I don't know a lot about DDD but I'm listening. I don't see a way for us to be able to validate that our relationships are not violated without including more than one aggregate (i.e. two/or more event streams). Which is why I'm eager to have support for committing changes to two (otherwise independent) event streams as a whole.

mpseidel commented 8 years ago

Hey @leidegre,

It think there is a conflict between

What we're trying to do is to create a group of hotel buildings.

and

we didn't want to invent the concept of a group

It seems that you already have defined the concept (you have described it perfectly in the second paragraph) it is just not reflected in your code/data design. This can make things hard going forward as you need the concept in the reporting context.

But the concept of a group or hotel complex does not have to be everywhere. You might be able to just intruduce it where you need it.

Anyways, as I understand you, you need the information in a reporting context where you report on single hotels most of the time? If a hotel is a hotel group than the numbers are being aggregated from its children right? Can you elaborate what the reporting looks like in practice?

leidegre commented 8 years ago

OK, @mpseidel

The reporting is just aggregations in a data warehouse. Presently all they have are hotels but now that we can group hotels they can use those groups to create aggregate reports based on a group of hotels. It's just something to group by in your SQL database, it's really nothing fancy.

Help me understand because I don't see how it's possible to "design" away the fundamental aspect of having a relationship between two entities and at the same time validate that this relationship isn't violated.

rkaw92 commented 8 years ago

@leidegre: Do hotels contain hotels? Your Aggregate Roots need to be as close to the truth as possible. I find it hard to imagine a "matryoshka" hotel, therefore I second the suggestion that having groups is the way to go. However...

A somewhat non-obvious solution that I sometimes use in my entities is having a reference (in your case, a "group ID") to something that does not really exist. This way, you are creating grouping without actually defining the Group entity. Think of it as tags. Then, when you need to add a hotel to a group, you simply re-use a tag that some other hotel has used. You are free to choose whatever style of identifier you like - even UUIDs should be a nice fit.

Then, using your Reporting Stores, you could present the user with a choice when creating a hotel - whether to assign it to a pre-existing group (where you display a search box or a drop-down of existing groups, which are name-less but are recognizable by the human by their members only - so essentially they look a bit like sets that are ValueObjects!), or to make up a unique group ID just for it. The advantage of such "lazy design" is that you can always materialize the concept into an Aggregate Root and you already have the IDs in place.

In such a system, there is no way to violate the relationship that you mention, because there is no relationship to begin with. You were never trying to model a parent-child system, in my opinion, and the entire "hotel-hotel" relationship is just a side product that is best avoided.

leidegre commented 8 years ago

@rkaw92 thanks for your insight. Do you have any other examples of how you can model relationships? And specifically how to maintain consistency. This whole issue is centered around the need to commit two distinct events atomically to be able to say that the relationship was created without violating some of our validation logic.

mpseidel commented 8 years ago

@leidegre I think the only really clean way to do this would be to model this problem inside one aggregate. Thats the whole idea arround consistency when using eventsourcing and ddd (http://geekswithblogs.net/Optikal/archive/2013/04/07/152643.aspx).

You can always try to enforce consistency by locking stuff, use transactions and other hard to deal with stuff. You can have a look at the saga pattern or process managers.

But well thought out aggregates can make your life so much easier because you dont have to care about consistency enforcement between multiple aggregates.

To me it sounds very much that "the relationship was created" should be one event and not two. In real life you don't have a marriage or a parent-child relationship happen in two different places that then have to be coordinated to make it consistent.

But maybe I am missing other options that would solve your problem. Maybe you can also reevaluate if an eventstore is even the right tool for the job here.

Maybe it would help to help if youd describe some more of how your current aggregates look like e.g. what functionality they contain. Really would love to help you out here :)

rkaw92 commented 8 years ago

@leidegre The usual way that I have employed to model real parent-child relationships is this:

Have a parentID in the child, but no list of children in the parent
Validate parent's existence in a Domain Service that performs 1 read from the parent's Aggregate Root and 1 write to the child
Make use of the fact that Aggregate Roots may not be destroyed in Event Sourcing - if we've validated the existence once, it will always be valid

The real trouble begins when you need to do something to all children of a parent, in the domain. That, however, is another matter.

susannaroden commented 4 years ago

Closed due to inactivity for multiple years.

thenativeweb / node-eventstore

Feature request, commit multiple event streams in one go #63