biothings / biothings.api

BioThings API framework - Making high-performance API for biological annotation data
https://biothings.io
Apache License 2.0
45 stars 25 forks source link

Consider separating scheduling as an individual feature #169

Open namespacestd0 opened 3 years ago

namespacestd0 commented 3 years ago

Currently, many actions that trigger subsequent actions define these automations within the preceding action. This pattern sometimes breaks the feature isolations, and make automation management spread out across the application. When encountering errors, it can also be difficult to define the retry strategies because of the lack of a centralized component that tracks the scheduling, and execution of tasks. The stated issues come from real-world use cases, but are also rudimentary, and deserve more deliberation of implementation details.

zcqian commented 3 years ago

Typically a message queue is used to schedule things like these. RabbitMQ, ZeroMQ, or Apache Kafka (technically not just a message queue).

Using a message queue can greatly decouple the components and can be very helpful in many ways down the line, but such architectural change is pretty complicated.