javiertoledo
commented
10 months ago Hi @josecelano, thanks for sharing your thoughts on the work done here and your perspectives. It's greatly appreciated!
Event sourcing and CQRS are great for any situation in which you want to keep a full trace of the changes, be able to "rewind" your state, keep the application receiving requests no matter what, and easily recover from errors after the issues are solved. If you think about it, these behaviors are desirable in all applications. Still, the cost of implementing event sourcing at scale has been historically high, and the cognitive complexity of CRUD is low enough to make most people think they're saving time by avoiding CQRS+ES or that applying event sourcing is over-engineering or a waste of storage space.
That's what we tried to solve with Booster: we wanted to make CQRS+ES as easy as writing regular CRUD applications, hiding the complexities of dealing with event streams, database sharding, event ordering, eventual consistency, and all the other fun parts of doing event sourcing at scale. With the current version of Booster, we've greatly reduced the initial learning curve with a convenient set of abstractions and default infrastructure implementations based on serverless containers and databases. But we haven't fully removed the cognitive complexity of having to "think in events" yet.
In order to switch, you need to stop thinking about designing around resources and designing user interactions as data model mutations. You need to start designing and thinking about the facts you want to store (events) and then build your data model on top of them, but most people have learned to do it the other way around when using CRUD frameworks, so when they try Booster, they end up building a CRUD system on top of a CQRS+ES system, and introducing a lot of code redundancy (with a command that accepts a whole entity schema, an event that just transports the data to the entity, and a read model that is just a copy of the entity.
You also need to learn to feel comfortable with eventual consistency. As you mentioned, most people puzzle with the fact that they don't have access to the projected read model right after the write, so you have to redesign your UIs to handle this correctly. Most APIs are designed to be synchronous, making users wait until the whole process is completed. In CQRS+ES, you respond right after passing validations and creating the events. Still, you don't get the computed read model immediately, and you need to handle that with your UX flows using optimistic updates or creating flows that focus on the events instead of managing entities (for instance, you have a form to submit an order, and then you reach a screen that shows the order as submitted with the data you sent, but a "pending" state, until you receive the read model and fill the missing data)
Event sourcing has clear advantages, and as someone who has used it in production, I can tell you that it has saved the day on more than one occasion, but it's still hard to implement in most scenarios, which clearly limits the adoption. The thing is that once you pass the initial struggle and learn how to use it properly, you find that, as it's shown in the study in this repository, the complexity doesn't disappear but is moved to a different place that allows way more decoupled designs, making maintenance much more approachable.
I think there are still ways to simplify and make these patterns more approachable, but there's still a lot of work to do to design the right set of abstractions and infrastructure in the right package that makes it really easy to use from minute 1. I'll personally keep trying as we evolve Booster, but I'd be happy to see anyone else solving this in a different way. The world deserves more event sourcing! 😄
orefalo
Hi @javiertoledo, I like this repo, the READMEm and the deterministic and analytical way to reach the conclusions. Congratulations!
Some time ago, I was looking for this type of study, but I did not find any.
I was especially worried about how often developers consider the CQRS+ES solution "overengineering". In my experience, when you have to maintain a project after prototyping you have to do a lot of "overengineering" in order to make it work. And complexity grows exponentially.
I also tried to compare both solutions (without Event Sourcing and without metrics). I just wanted to compare the readability and maintainability of both solutions.
As you mentioned, you should evaluate if the solution fits your case, but something I see very often is that applications that are intensive in reports/views/read become very fast a chaotic mess, where maybe you can have a well-organized code, but where all is coupled to all in the database. Very often, all the complexity is hidden in the database, and the domain logic is very coupled to the persistence. I guess you could have projections (read models as DB views or duplicate info in other tables or DBs) without CQRS or ES, but I think it would not be as easy to integrate as when you use these CQRS+ES patterns.
On the other hand, I have to say I have no experience with this approach in production and I know there are other problems you do not have with the CRUD version. I would like to see also a comparison between things that are easier but also with things that can be harder. In my case, for example, after executing a command you do not have the projection of the new entity immediately and you have to deal with that type of asynchronous problems when you split the write and read models.