Determine key considerations when selecting a messaging platform

[seallard]

What are the most important aspects of the platform? What aspects of it should be considered? Use case: basic pub/sub pattern. Scale: small, < 1 million events/month long term. Hosting: on-prem

[seallard]

• Cost
• Hosting: Cloud-based vs. on-premises solutions.
• Operational Complexity: Expertise required for setup, maintenance, and scaling.
• Reliability in Operation: Uptime, fault tolerance, redundancy.
• Long Term Stability: Maturity, community support, organizational backing.
• Ease of Integration: Quality of client libraries, SDKs, APIs, compatibility.
• Supports Pub/Sub
• Documentation: Comprehensiveness, ease of use, support resources.
• Delivery Guarantees: At-most-once, at-least-once, exactly-once delivery.
• Ordering: Message ordering guarantees and configurations.
• Persistence: Message persistence, retention policies, durability.
• Observability: Ease of monitoring and debugging

[samuell]

Some random ideas for aspects to consider:

• Pluggability - Ease of adding new components and pipelines without changing the internals of components
• Observability - Ease of monitoring and debugging
• Message delivery guarantees (at least once, at most once, exactly once etc)
• Ease of resolving data inconsistencies manually/semi-manually

[seallard]

What we need

• Cost: preferably free.
• Hosting: on-prem.
• Operational Complexity: easy setup and maintenance.
• Reliability in Operation: Uptime, fault tolerance, redundancy.
• Long Term Stability: mature, certain to be around for a long time.
• Ease of Integration: need existing Python clients.
• Supports Pub/Sub
• Delivery Guarantees: at-least-once or exactly-once delivery. We can shape the architecture to fit any of these. I think ensuring idempotent event actions and having at-least-once makes the most sense.
• Ordering: We would likely benefit from being able to guarantee message order.
• Persistence: Need persistence of messages for DLQ or if broker/consumer goes down.
• Observability: need to be possible to integrate with some monitoring solution.

[seallard]

Lets say we have 20 asynchronous steps in our organization for each sample.

1. Order placed
2. Reception control
3. Lab prep
4. Sequencing
5. Data transfer from sequencing machine to NAS
6. Demultiplexing
7. Demultiplexed data transfer from NAS to Hasta
8. Post demux processing
9. Sequencing QC
10. Job fabrication/analysis
11. Post analysis QC
12. Post analysis processing (concatenate fastqs for microsalt might be the only one?)
13. Data upload
14. Customer notification
15. Encryption
16. Compression
17. Archiving
18. Cleaning

Lets say we process 5000 samples per month and that each step in the flow is per sample (not true). Then our load would be, at most 100 000 events/month. That is nothing.

Even if we started processing a lot more samples and extended the use of events across our services, I don't think we would hit more than 10 M events/month.