Prooph Event Sourcing Revival

[codeliner]

Two years ago I created this issue about a possible Blueprint instead of a event-sourcing library.

I'd like discuss this option again. In the last two years @sandrokeil and me focused on developing a modeling tool: InspectIO. It ships with a customizable code generator. The PHP version works with PHP AST and @sandrokeil put a lot of effort into making it as flexible as possible while still being easy to use.

With this toolbox in place we could think about a new prooph/event-sourcing and also a lightweight prooph/service-bus. Both based on code generation instead of library code that you have to extend from.

A set of defaults could guide people in the right direction but should offer enough freedom to change things they want to do different.

What do you think?

/cc @prolic

[prolic]

While I am not totally against having an event-sourcing lib available, this would be different for v7 and v8 of event-store and again different within prooph/micro f.e.

What is the purpose of having it as library available? Rely on it during code generation?

About: prooph/service-bus What we have is already solid and reliable. What else do you want to remove or add? Any featureset in mind?

[codeliner]

I'd like to turn prooph/event-sourcing into a library that is preconfigured for the code generator. See this event-engine package as an example: https://github.com/event-engine/php-code-generator-cody It's prepared to generate Event Engine code.

We could prepare different styles for prooph/micro, v7, v8. Some logic like command, event and value object generation can be shared across the styles and others like organizing aggregates, command handlers, etc. is different. some details: https://github.com/event-engine/php-code-generator-cody/blob/master/src/Aggregate.php#L58

---

Regarding service-bus: we deprecated the package because maintenance was too much effort and other feature-complete solutions exist as alternatives. So I'm thinking about a very simple, basic implementation: map command to handler, event to listeners, without all the plugin logic etc. Just enough to have something that works out-of-the-box when exploring prooph and its capabilities.

If people need more they should follow the suggestion and use symfony/messenger or whatever they prefer.

[fabiocarneiro]

I would +1 on reviving this (excluding the code generation part).

I have been trying to revive a project that was stale for the last 5 years and the stack was with Prooph Eventstore on Mysql, projections on service bus 5.x (listeners with onEventHappened methods), Zend expressive for API communication with REST, HAL, ApiProblem...

The goal was to touch as little code as possible, but I managed to upgrade almost everything (k8s, Prooph, Mezzio...) to the latest versions and as data wasn't a concern (and other reasons), swapped the Prooph event store with Greg's Eventstore. I wanted to run everything in a single process, but as the async Eventstore client does work with AMP (and Mezzio with Swoole), I opted by splitting the projection code in its own application. The writes are happening with the HTTP library.

Almost everything is working fine but I'm a little bit stuck for days in the projection part (it works but slow and unreliable). I made a "compatibility trait" with call_user_func on _invoke to make my old projection classes written for 5.x work on 6.x. In general, I would favor small callables instead of code all merged together, but in the case of projections, I still prefer the old approach of 5.x since its much more similar to the code written by the query/projection system of Greg's Eventstore (you receive state + event and return the new state).

Since MySQL is still used for read-model projection, the challenge is to batch event handling to avoid doing a database commit on every event. On the projection code, I obtain the event from the event store client with a persistent subscription and pass it on the event-bus from service-bus. Since the event bus is fire-and-forget, it is hard to know when the event was successfully processed so the manual acknowledges can be sent to the Eventstore. I know the library is not made for that, but message brokers often work with acknowledging patterns, so having a retry-system in the service-bus could also make sense.

I'm not aware of a feature-complete alternative as you mentioned above. I did look into symfony/messenger, but I had the feeling it is a bit coupled to the framework or it would be a small hassle to configure it.

In my humble opinion when you talk about code generation you talk about an opinionated code that provides no guarantee of being easier to modify/customize than a properly contract-based design. Your mind is less shaped with the positive limitations that come from knowing you are providing something in a library.

I know you guys went in a completely new and opposite direction with a new framework, and RAD is probably much faster, but not everyone is a fan of code generation or can just switch the approach on an old app as I have. For the purposes that fulfill the other framework, why don't you guys write new packages in that repo instead of trying to transform Prooph into something else?

[prolic]

Having thought about this a few more days...

event-sourcing lib

Actually I am quite happy this thing is gone, most devs I know didn't use our implementation anyway. If you want to work on this, I have nothing against. I can do some code reviews and provide input, but most likely I personally won't invest too much time into this.

service-bus

This is actually quite interesting. Last stable release is 6.2.2 from Feb 7, 2018. 3 years without a single bug-report. Also this thing has more downloads then event-store v7 and v8 combined, so people really seem to have an interest in this project. In my current project for client I use simple anonymous functions for this job, instead of composing and wiring many objects together (like the prooph/service-bus does). I am curious what plans you had in mind and how code-generation can be used to help build a service-bus component.

[webdevilopers]

FYI @psren and @OskarStark were recently discussing the symfony/messenger as standalone-version:

• https://twitter.com/psren/status/1362870629570142209

Maybe @gquemener could also provide some feedback.

[gquemener]

Haven't got much experience using symfony/messenger as a standalone component. All I know is that it is designed for such purpose (and is integratable within a symfony app through the symfony/framework-bundle package).

However, I guess you'd have to do quite a bunch of source digging to properly configure the bus manually.

Nonetheless, I was able to achieve the following feature without too much hastle with symfony/messenger and the framework-bundle:

• register a command bus and an event bus
• wrap command handlers within a doctrine transaction
• dispatch domain events after command handling and transaction commit (through a middleware, so this behavior is configured within the command bus, and generalized to all command handling)
• route some events to a message queue (through the use of a marker interface)*

and as a bonus I had : log and good DX through the web debug toolbar (where I see all the command/events that have been dispatched/handled during an http request handling).

*It is important to know that messages routed to async transport will never be handled synchronously. If you want your message to be both routed to a message queue and your local event handlers to be called, you must explicitly route your event to both sync and async transport.

[codeliner]

@fabiocarneiro thanks for your insights!

I'd like to answer a few things:

In my humble opinion when you talk about code generation you talk about an opinionated code that provides no guarantee of being easier to modify/customize than a properly contract-based design.

It's the opposite: I'm talking about a blueprint powered by code generation that serves as a default implementation. Of cause the blueprint is opinionated, but compared to a library you can tweak it to your liking.

For the purposes that fulfill the other framework, why don't you guys write new packages in that repo instead of trying to transform Prooph into something else?

That's already done. I don't want to turn prooph into a RAD framework. Therefor we have Event Engine and it is already powered by the Code Generator. What I suggest here is to integrate the benefits of visual modeling + code generation into prooph, which calls itself enterprise-ready. getprooph.org still lists all components, but everything except event-store is not officially supported at the moment. I would like to change that again, but still respect the reasons why we deprecated the stuff two years ago.

[gquemener]

I should also point out that symfony/messenger seems to lack a way to enforce a single handler per command strategy : https://github.com/symfony/symfony/issues/36958

[HellPat]

I was asked on twitter to leave a comment here regarding standalone usage of symfony/messenger:

I should also point out that symfony/messenger seems to lack a way to enforce a single handler per command strategy : symfony/symfony#36958

Since handling the messages is done by a Middleware, it should be easy to build an own SingleHandlerMiddleware implements MiddlewareInterface and use it instead of the default HandleMessageMiddleware, or you could even use the default HandleMessageMiddleware and provide a custom HandlersLocator which only ever returns one handler.

I did look into symfony/messenger, but I had the feeling it is a bit coupled to the framework [...]

It's not coupled at all. It's possible to use it standalone and it is very feature-rich and has lots of transports, including:

• https://packagist.org/packages/sroze/messenger-enqueue-transport
• https://github.com/symfony/doctrine-messenger
• https://github.com/symfony/redis-messenger

However, I guess you'd have to do quite a bunch of source digging to properly configure the bus manually.

This is true. The docs for standalone usage is quite more than minimal (also see this docs-discussion on twitter), but the sourcecode is well understandable. I spend little time to get it up and running with a simple setup (https://github.com/HellPat/symfony-messenger-standalone/blob/main/bin/console).

I also experimented a bit with the standalone usage and can say:

• it's easy to extend, and every corner is replaceable
• eco system / integrations are good

I will continue to invest some time in this and really like symfony/messenger. The symfony-bundle is very easy to use, but everything seems doable as standalone, too.

But if you want async-command-processing I'd recommend using symfony/console, since the Console-Commands provided by the messenger-Component are huge time savers.

*It is important to know that messages routed to async transport will never be handled synchronously. If you want your message to be both routed to a message queue and your local event handlers to be called, you must explicitly route your event to both sync and async transport.

If you do want a message to be routed to multiple transports an own HandleMessageMiddleware could do both, too.

I know the library is not made for that, but message brokers often work with acknowledging patterns, so having a retry-system in the service-bus could also make sense.

retry-patterns is built in in symfony/messenger.

In my humble opinion when you talk about code generation you talk about an opinionated code that provides no guarantee of being easier to modify/customize than a properly contract-based design. Your mind is less shaped with the positive limitations that come from knowing you are providing something in a library.

Serialize/Deserialize is completely configurable with symfony/messenger so using generated code should be no problem at all. (I'm a fan of generated code).

Maybe one could provide a simple factory that builds a "opinionated" messenger instance.

[fabiocarneiro]

I was asking if the guys from Eventstore had an example of how they handle the persistent subscription in other languages and they unfortunately also don't have examples but have been planning to work on it.

The interesting part is that Greg also answered and said that there were multiple requests for a client-side projection library and they have been planning to work on it for a while. I quote:

I have discussed with a few others in the past sitting and writing an external projection library. The first would be in C# then ported to Java. This is a 4-8 week project depending on scope likely involving 1-2 people aside from me personally. I would love to get some companies involved so its not just "free work". There is no shortage of time in COVID days

[fabiocarneiro]

It's not coupled at all. It's possible to use it standalone and it is very feature-rich...

@HellPat, I might not have expressed myself properly. I didn't mean you must use or download the other components, but it works in a very standard Symfony way. The way the configurations work... Bundle...

[codeliner]

thanks for all the input guys!

I think we have two topics here:

1. Event Sourcing @prolic yeah, I get your point. I think it is best to provide a prototype of what I have in mind and collect some feedback. I'm also happy that it's gone because everybody has a different opinion about it. But what I also learned during last 2 or 3 years is, that companies are really looking for proper Event Sourcing solutions that they can integrate into their existing stack. And prooph components are a perfect match for them because our components are really individual packages, but still work quite nice together. No matter what part you develop yourself and what part you integrate directly: prooph/common is the glue. I'd like to provide an easy way for them to start with a blueprint and then learn and adopt. I could do that outside of prooph under the Event Engine umbrella, but IMHO prooph OS is the better place for it, because that is what people know and where they are looking first for solutions. I don't think you need to develop anything, but feedback and some code review would be awesome, yes!

2. Service Bus I remember the day when we made the decision to deprecate prooph/service-bus. It was a hard one. The package is of extrem high quality. And this ongoing usage after 3 years of inactivate still demonstrates it. Symfony/Messenger has a slightly different scope but is also of high quality and can be tweaked to provide similar capabilities like prooph/service-bus. Personally I also think that a very simple message routing solution is enough for many cases, especially these days where more and more infrastructure stuff is directly pushed into the cloud and application logic split into micro services or even single functions. I'm really not sure about how to proceed here. I'd like to have something in the prooph ecosystem that gives you message routing. Again because people are looking at prooph for a solution and they don't really get what packages are still maintained and which one are not. Here my suggestion is also to start with a new prototype and collect feedback. Or we just reactivate prooph/service-bus and see if we can put a light version next to it. Not sure ...

[codeliner]

Update from my side:

@dgafka asked me to take a look at his EcotoneFramework before reactivating prooph/service-bus. Ecotone is a service bus implementation which enables message driven architecture and CQRS/ES. We had a longer chat last week and decided to collaborate on some ideas like InspectIO Code Generation for Ecotone, proper prooph/event-store integration and maybe incorporate some ideas from Event Engine.

Ecotone is definitely as powerful as prooph/service-bus but easier to understand thanks to PHP 8 attributes. prooph/service-bus had this plugin system based on an internal event dispatcher. Ecotone on the other hand uses the concept of interceptors which can hook into message routing and provide cross-cutting functionality. Compared to symfony/messenger Ecotone has first-class support for CQRS which was another selling point of prooph/service-bus before.

Long story, short: With Ecotone being actively maintained and making use of latest PHP 8 features, it makes no sense to me to reactivate prooph/service-bus. Ecotone also provides an Event Sourcing module, so I think that's a very good option for people looking for a library that provides ES basics.

I'm closing the issue here. Thank you again for all the input!