AdrianHu99
commented
6 years ago they are doing things to deliver events to different endpoints by different APIs. First: one worker responsible for all endpoints -> too heavy, backpressure. Second: one worker per endpoint -> for one pipeline, if one specific type is dominant, then how about the other events? How to design the queue? Third: queues per <source, destination> -> so many different queues! Fourth: Use database to store execution order! Fifth: Use DIRECTOR to manage caching, locks, and querying in process, it works with Consul to do the work. They have many individual directors, each handling a small slice of traffic.
https://segment.com/blog/introducing-centrifuge/