Discussion: async policy handling

[TomKaltz]

Since policy events are handled one-by-one from the all stream, there is a possibility for a policy to severely lag behind if slow queries are used in the policy handler. My current solution is to have the policy add a job to a BullMQ queue using an external system artifact. This results in a lot of boilerplate code just to make policy handling performant.

It dawned on me that this sort of async parallel policy handling might be a candidate for a core feature or plugin for eventually. The general idea is that a policy artifact could have a configurable flag to allow queued/async handling instead of the outbox/serial/checkpoint style handling. The subscription store will mark what events were successfully put on the queue system and the queue system would handle redelivery, dead lettering, etc.

Does this type of feature sound wildly outside of the scope of a framework like eventually? Is there a huge gotcha I might be missing?

[Rotorsoft]

This is a topic I was planning to tackle next, but haven't found the time to focus on it. It's basically the concept of subscriptions by correlation instead of at the artifact level, so you can only serialize flows that are strongly correlated. This should solve the scalability issues you are describing, but it requires some work at the broker level to understand new metadata to auto configure the system.