search

proophsoftware / es-emergency-call

Struggling with CQRS, A+ES, DDD? We can help you!
BSD 3-Clause "New" or "Revised" License
26 stars 0 forks source link

How to model master data / tenant specific models #8

Open pmartelletti opened 5 years ago

pmartelletti commented 5 years ago

Hi @codeliner ! I know you guys are busy right now with the fee office example (#7) , but just thought I'd place the question here, explain my domain and then, whenever you guys get a minute to review it, we could go over it. 😄

So, a bit of a context from our company here: we're a company that builds software for grocery stores, most of the hard work is done for the POS in the actual store and in the back office. The POS is built in .NET, and currently being rebuilt using CQRS.

My team, however, is focused more in web applications that support our customers, like transferring files from store to head office, place automatic orders, sync products and offers between the different head offices (as the main products base and offers is the same for all customers). We use PHP (mostly Laravel) to build our apps, and JS to connect to the API's we generate.

The project we're currently working is the offers one. This is already working, the customers are happy with it. However, a new requirement was added to the system, and things started to get messy: because we didn't have a log of what was happening and when in the system, we were unable to respond questions from the customers like "when was this item introduced to this offer" or "when was this offer extended". That's when we decided to stop what we're doing a try to come up with some ideas on how we could implement this. Use Laravel log could have been enough for the actual questions/reports we have, but that wouldn't help us for any future questions customers will have. That's why, after hearing the other team talking about ES all the time, we decided to give it a shot.

=======

Now, some background on the Domain and what our app needs to do.

All our customers receive daily a text file saying which offers are active (or will be), and what items are in that offer (they are all part of a cooperative group, so basically all members of the group receive the same offers, as the group makes the purchase for them). If they use our EPOS system, then we offer this added service where we parse the file and sync their Head Office with the deals, automatically - so no human interaction is needed. If they are not using our system, then they'd have to manually key in the changes daily.

The domain was easy enough when all customers receive all the same deals - however, it started getting complicated when the new requirement was added: customers would like to be able to ignore deals they are not interested to offer, to create their own deals, or to amend deals (that is, if the deal was buy any soft drink for €1, but the group said only Coca-Cola brand was in the offer, there might be a store that wants to add Pepsi brand as well).

That's were it all got complicated - we haven't found so far any Domain that talks about "Master Data" and "Tenant" specific data. The aggregate started to look really ugly when we added flags like "this offer is specific to this tenant", or "this tenant is ignoring this or that offer".

So here's where we can to your help. Can you help us try to get the correct models for our domain? Here's a bit of example of what the system should be doing.

That's essentially what we had before. The aggregates were easy to understand, it was all straightforward.

However, the things got more complicated with these new requirements, as customers should be able to say:

So, with all these new rules, it all started to get complicated and we're kind of stuck, or slowly making any progress. So wanted to know if you could help us:

a) Try to understand how tenant data would work, as I've never seen an example for DDD like that b) How would you handle the ignores or the custom data? Where would you keep track of that?, etc

Also, we really need to have a track of the import process, what was imported and when, and what failed, what was skipped because of the customer settings, etc, as we usually get questions like "why is this item not in my deal, but it is in the other customer?". Would you do that in DDD as well?

Thanks a lot for your help in advance - and hope the problem is clear. I'll more than glad to help clear out any questions you may have about it.

codeliner commented 5 years ago

Hey @pmartelletti , very interesting problem space ;) thx for sharing.

You're right. We're really busy at the moment. We joined a developer team to help them with their core system. It's an enterprise business and they are currently in trouble so we have a lot of work on our desk. We need to understand the system, identify bottlenecks and remove them. Then we need to split the system into smaller units and stabilize it. This means that the next months (probably until February/March 2019) we can't spend much time for OSS. My plan is to find a bit time every week to continue with the Fee Office. We're working on it since a few weeks and collected a lot of information, but @enumag 's team still needs some answers/guidance.

That said, I don't think we can go through your problem this year, sorry. But maybe some other ppl in the prooph community can help. ping @gquemener ?

Anyway, you can read through the Fee Office issue and see what we did there. I'm currently summarizing the issue here: https://proophsoftware.github.io/fee-office/ddd/

In short:

Do an event storming session with your team and share the result on public board on https://realtimeboard.com/

Do the Event Machine Tutorial: https://proophsoftware.github.io/event-machine/tutorial/ (takes 2-4 hours) and start working on a prototype using the Skeleton: https://github.com/proophsoftware/event-machine-skeleton

Make it a public repo and invite ppl to help you with the design. I always have 15 min during a break or so to give some advice, but the next emergency-call has to wait.

pmartelletti commented 5 years ago

hey @codeliner - thanks for your reply! Yeah, I suspected that. Because we do have a deadline, we started working on the code, even we didn't like how it turned. Most probably the end result will be fine, and will work, however, we still feel there's stuff that can be improved so that's why I came to ask for help, maybe for a second iteration of the development (or some refactoring!).

So, what I can do is work during this weekend on a prototype using the Skeleton, make it public and share it here, and maybe when you have 10/15 minutes to have a look, you or any other member of the community could tell us if we're going in the right way, or if we're really complicating ourselves too much and things could have done easier / better.

Does that sounds good to you?

Thanks again!

codeliner commented 5 years ago

One advice for your real world project: Try to set up a module system similar to the one I've explained here: https://proophsoftware.github.io/fee-office/intro/module_system.html#1-3

Not sure how to do it with Laravel, but I'm pretty sure that it is possible. When you have those modules and follow the rules then it's easier later to throw away some of them and replace with a better implementation. Greg Young did a great talk about that topic: https://vimeo.com/108441214 Try to design the system in a way that you can easily delete code. That's a secret weapon to fight the unknown unknowns

enumag commented 5 years ago

This is quite interesting topic. I'll definitely keep an eye on it because we will need something similar in our system as well eventually.

@pmartelletti As you already know I'm not an expert but let me share my thoughts anyway. I think I'd go with something like keeping the master offers as templates only. Each customer would have a set of rules how to use the master offers. Then there would be some ImportOffersProcess aggregate that would take the master offers and customer rules to create customer offers (for one customer). The process would keep a track of what it did with each master offer - if it got changed or ignored and which rule was used to do that, answering your questions "why is this item not in my deal". Is this something close to how you're approaching it? What are the problems with this approach?

pmartelletti commented 5 years ago

hi @enumag thanks for your interest in our case :smile:

Well, to be honest, we though about something like that at some point, but we would have ended up with 2 aggregates that were identical, one for the master data, and one for the specific / tenant offers. So we decided to have one OfferAggregate, with an optional flag saying if that was tenant-specific offer (a tenantId, basically), or if it applied to everyone.

Then, on the read model, we'd generate an entry per tenant for that given offer (if the offer was for everyone), skipping the tenants that had any rule to ignore it.

So basically, the 'ignore' part was done on the projection, which I'm not sure if it's 100% correct, thus why I came here asking for help :laughing:

Makes a bit of sense what we're doing?

gquemener commented 5 years ago

Hello :wave:

In term of responsibility, I believe the Mother Company will always be the one sending offers on a daily basis to the customers. It's up to the customers to ignore or not these offers, based on their own sets of rules.

So I would consider a customer (is this how you refer to them in your company?) has an aggregate gathering, among other things, a set of rules which will dynamically define which offer has to be accepted or rejected (just inventing the terms, again the question is how this process is referred to from your business expert perspective).

What do you think?

pmartelletti commented 5 years ago

@gquemener thanks for joining the conversation 😄

Yup, that's exactly what we have for now. The question is more: how do we store the offers? Should we keep a record of all the offers that were sent from the Mother Company (in case, I don't know, a new customer is added?)? Should we just keep Offers that the Companies accepted (with a reference to the company id in that aggregate?) Should they be 2 diff aggregates?

That's what our main issue here. How to avoid duplication by creating 2 diff aggregates that hold the same properties, but to the business, they represent 2 different things: one is a "template", one is the actual offer that will be sent to the customer.

Makes more sense? 😄

gquemener commented 5 years ago

Yep it does.

Although they look similar, they are not the same thing AFAIU.

I kind of see two contexts, a "Mother Company" context in which an offer represents something, and a "Customer" context, in which an offer represents the same something, plus the fact it's been accepted/process/handled or rejected/ignored (the exact term is the one that's used by your customer!).

It's important to see that this two aggregates will have their own separate consistency boundaries. If you were to reference a mother company offer within a customer offer, then the acceptance/rejection of the later will lose its sense/consistency if the mother company modifies the content of the offer. That's why I think it is important to duplicate this information.

As you say, a Mother Company Offer is a template, which solely is useful for Customer Offer creation purpose.

Should we just keep Offers that the Companies accepted

Which term do the experts use? Company, Customer, Client?

pmartelletti commented 5 years ago

Sorry about the mixup, we use customers. So basically, we have templates for the Mother Company, and actual offers accepted / rejected by our customers.

So, how would you handle duplication then? Let's say I add a new property to the template aggregate - that would mean I'd have to manually copy that details in the Customer Offer aggregate as well, Including business checks?

I can't think of the Mother Template being a simplified version of the Customer Offer, as the checks for doing actions (adding items, adding groups, change start dates, etc) should be the same in both. That's what worries me the most - having to duplicate all these domain logic that's inside the aggregate and, if they are in 2 separate bounded context, it would be duplicated (and there's a potential issue of forget adding / removing / changing in one of them 😟 )

gquemener commented 5 years ago

In my understanding, the template is solely used at customer offer creation time. Subsequent template modification wouldn't impact already existing customer offers, but I may have a bad understanding of your domain.

I think this is the time when an Event Storming session might become handy. Are you familiar with the concept?

gquemener commented 5 years ago

and regarding offer implementation, I would use something close to the EAV pattern (which allows to add any number of attributes with different values to a given entity/aggregate) => Open to extension / close to modification. But it'll depend on your business requirements :)

pmartelletti commented 5 years ago

In my understanding, the template is solely used at customer offer creation time. Subsequent template modification wouldn't impact already existing customer offers, but I may have a bad understanding of your domain.

That's the problem. If the template receives an update, then it should spread to all the customers that in their settings want to keep the offer in "sync". By default, this is the case - so it's the most common scenario. A modification in the template wouldn't affect the Customer Offer if they've done a modification to the offer, for example. There are some other rules (posted in my first comment), but I think you get the idea now.

We did an Event Storming before actually starting to implement this. And we came up with this model, but to be honest, I was the only in the team with knowledge on the domain and Event Storm, so basically, I could have "contaminated" that Event Storming session with my initial implementation ideas. 😟

gquemener commented 5 years ago

The context map is pretty straightforward in my head:


 +----------------+          +----------+
 |                |u        d|          |
 | Mother Company +----------+ Customer |
 |                |          |          |
 +----------------+          +----------+

PS: Please note that I'm quite experiencing and learning here, so don't take everything I say as correct! Let it be a start for further discussion :)

gquemener commented 5 years ago

so basically, I could have "contaminated" that Event Storming session with my initial implementation ideas.

That happens often, and it's normal as we've been taught to think like that! Do not worry about implementation, it will naturally flow from the domain model ;)

Let's take the TemplatePropertyAdded event. If this event needs to be propagated through Customer Offers that match some conditions, then I would produce this exact message to the outside world (meaning outside of the Mother Company bounded context) through a message queue for instance. Then there would be one (possibly scaled horizontally) consumer on the "Customer" side that would propagate the change given its own set of rules.

WDYT?

gquemener commented 5 years ago

Theorically, this consumer is what we call a Process Manager, because it listens to some event (TemplatePropertyAdded for instance) and dispatch command(s) if necessary.

Here it would fetch all offers that uses the template, check if they need modification and dispatch AddProperty commands if that's the case.

pmartelletti commented 5 years ago

@gquemener It kind of make sense I think, yes.

So, in the Process Manager, you'll fetch all the customers, and for each one, check the settings, and if the template change should be propagated, you'll dispatch a command to do that?

My only concern then is - where should Offer logic be, then? As, there are some rules in adding items to an offer, for example. Those rules would apply for both the template AND the customer offer, and ideally, I don't want to have to write them twice.

Apart from that, I think it kind of make sense what you're suggesting: it would be the tenant aggregate the one deciding if a change from the template should be propagated to the customer's offer. Saying that, how would you reference something in the template to an actual customer offer? Would you keep a "templateId" or similar?

gquemener commented 5 years ago

My only concern then is - where should Offer logic be, then? As, there are some rules in adding items to an offer, for example.

I would not focus on where to put logic, but more on how the system reacts to some events (through an event storming). The implementation will naturally flow from these processes. For instance, what happens to the Customer offer acceptance status once it is modified because the template has been modified?

how would you reference something in the template to an actual customer offer? Would you keep a "templateId" or similar?

Yep, that's what comes to my mind.

pmartelletti commented 5 years ago

I would not focus on where to put logic, but more on how the system reacts to some events (through an event storming). The implementation will naturally flow from these processes. For instance, what happens to the Customer offer acceptance status once it is modified because the template has been modified?

We had all those cases analysed during the initial brain storm, and that's why I know that the logic needs to be in both places.

Say, an item was added to an offer template. Then, the customer amended that its version of the offer, adding an item that was not in the original template, but chooses to continue receiving updates from the template. That same item is then added to the template, so the customer will receive that update: if we don't do the check at customer level, then the item would be twice in the offer.

Hope this bit makes sense to you, as I feel like I'm going in circles trying to explain it haha.

Apart from that, I think everything else is pretty clear and I'm ready to work on the prototype so @codeliner can have a look later.

gquemener commented 5 years ago

That same item is then added to the template, so the customer will receive that update: if we don't do the check at customer level, then the item would be twice in the offer.

It makes sense, but this specific business logic is not in two places, only at the Customer level, isn't it?

Hope this bit makes sense to you, as I feel like I'm going in circles trying to explain it haha.

That's part of the exploration process :D Providing examples is an excellent way to explain your concepts, thanks :+1:

Looking forward to check your prototype :)

pmartelletti commented 5 years ago

It makes sense, but this specific business logic is not in two places, only at the Customer level, isn't it?

Well, yes - but also at template level. I can't assume what the mother company is sending is always correct - I'd still like to do the same validations (like, item is not twice, you cannot add a reward for more than the actual price of the item, etc), because if I don't, when a new customer is added, and I have to copy the templates to actual customer offers, then.. I would have the duplicate stuff. Although that would be caught on the copy process by the Customer Aggregate, right? 🤔

Looking forward to check your prototype :)

It's all starting to make sense now, I think. Unfortunately our deadline is the end of the month and we're already deep inside the other approach: having one aggregate with customer id on it when it's specific for the user 😟

Anyways, I think we could use that model for delivery, but continue investigating this approach for a future refactoring, so still think it makes sense for me to build that prototype and publish it to Github some point this week.