sonus21
commented
9 months ago I was thinking about adopting the second option because if a problem occurs and maintenance is needed, this would not affect other functioning integrations.
This is 100% accurate and no one can deny this, but do we really need so many queues?
Two suggestion from my side
- Consider a prioritization approach. Perhaps creating five sub-queues, each independently handling its workload, could be a viable solution.
- Another strategy is to group low-throughput or similar sub integratrion messages into a single queue, simplifying the process of creating new queues.
It's crucial to assess the scale of your operations, including the volume of messages or events transmitted to each queue. Analyzing metrics such as averages, p95, and p99 can provide valuable insights.
Managing 575 queues might impose a significant burden on Redis, necessitating the addition of multiple Redis clusters to handle the resulting load.
It's important to recognize that there's no one-size-fits-all solution. Providing details such as message count, queries per second (QPS) at the queue or integration level, would aid in determining the optimal approach. Without a comprehensive evaluation of the options mentioned above, I would not recommend opting for 575 queues.
joeldatabox
I'm working on a project that needs to integrate with several different systems. This project will be responsible for collecting records generated by smartphones and passing them on to other client systems. With this in mind, I'm unsure what the best strategy is.
1st Have common queues for all integrations 2nd Have specific queues for each integration.
I estimate that we will initially have 25 different integrations and each integration will have the equivalent of 23 queues. Would it be better to have generic queues for everyone or to have specific queues for each integration? If I adopt a specific queue for each integration we will have around 25*23=575 queues.
I was thinking about adopting the second option because if a problem occurs and maintenance is needed, this would not affect other functioning integrations.