adamcharnock / lightbus

RPC & event framework for Python
https://lightbus.org
Apache License 2.0
199 stars 22 forks source link

Implement background tasks 2.0 #8

Closed adamcharnock closed 5 years ago

adamcharnock commented 5 years ago

The current background tasks implementation has the following issues:

  1. Lightbus processes for the same service will not coordinate. If a function is set to run every hour, it will run every hour for every process.
  2. Because the background task will run in a ThreadSerializedTask it will block other RPCs/Events from being handled (potentially leading to RPC timeouts).

I think the solving 1 will provide a fix for 2 (namely: running multiple Lightbus processes)

Proposed solutions:

  1. Designate a single process as the background task handler (simple, not able to handle many background tasks running concurrently, requires process designation)
  2. Designate a single process to fire a run_background_task event. All Lightbus processes listen for tasks on this event stream (more complex, can handle many background tasks running concurrently, requires process designation)
  3. Find an idempotent way of firing events (e.g. generate consistent message IDs). Looks like this will be a faff in Redis.

I actually like option 1 because:

  1. It would still be possible to have one process handle everything (good for small deployments)
  2. One could trivially fire events from within a background task, thereby allowing distribution of work manually to many lightbus processes via events

How could this process designation look on the command line:

# Worker and background pair
lightbus run --only=tasks
lightbus run --skip=tasks

# Other random examples
lightbus run --only=rpc,event
lightbus run --skip=event,tasks
adamcharnock commented 5 years ago

This is now done and merged