Need to design a mechanism that will schedule background tasks in a fair manner, allow scaling of worker processes, and be tolerant of worker failures, and have no single point of failure. The current design uses a single multi-thread process that will recover lost work on restart after a process failure (but not a node failure) and cannot scale to multiple processes. The task scheduling logic is in the database micro-service and does not consider maximum scheduled worker overall or per-user.
Need to design a mechanism that will schedule background tasks in a fair manner, allow scaling of worker processes, and be tolerant of worker failures, and have no single point of failure. The current design uses a single multi-thread process that will recover lost work on restart after a process failure (but not a node failure) and cannot scale to multiple processes. The task scheduling logic is in the database micro-service and does not consider maximum scheduled worker overall or per-user.