The current background tasks implementation has the following issues:
Lightbus processes for the same service will not coordinate. If a function is set to run every hour, it will run every hour for every process.
Because the background task will run in a ThreadSerializedTask it will block other RPCs/Events from being handled (potentially leading to RPC timeouts).
I think the solving 1 will provide a fix for 2 (namely: running multiple Lightbus processes)
Proposed solutions:
Designate a single process as the background task handler (simple, not able to handle many background tasks running concurrently, requires process designation)
Designate a single process to fire a run_background_task event. All Lightbus processes listen for tasks on this event stream (more complex, can handle many background tasks running concurrently, requires process designation)
Find an idempotent way of firing events (e.g. generate consistent message IDs). Looks like this will be a faff in Redis.
I actually like option 1 because:
It would still be possible to have one process handle everything (good for small deployments)
One could trivially fire events from within a background task, thereby allowing distribution of work manually to many lightbus processes via events
How could this process designation look on the command line:
# Worker and background pair
lightbus run --only=tasks
lightbus run --skip=tasks
# Other random examples
lightbus run --only=rpc,event
lightbus run --skip=event,tasks
The current background tasks implementation has the following issues:
ThreadSerializedTask
it will block other RPCs/Events from being handled (potentially leading to RPC timeouts).I think the solving 1 will provide a fix for 2 (namely: running multiple Lightbus processes)
Proposed solutions:
run_background_task
event. All Lightbus processes listen for tasks on this event stream (more complex, can handle many background tasks running concurrently, requires process designation)I actually like option 1 because:
How could this process designation look on the command line: