NicolasLM / spinach

Modern Redis task queue for Python 3
https://spinach.readthedocs.io
BSD 2-Clause "Simplified" License
63 stars 4 forks source link

Allow periodic tasks to define their exact wall-clock time to run #48

Open juledwar opened 3 months ago

juledwar commented 3 months ago

The periodics are currently effectively a 'time since it last ran, which in turn is time since it first ever ran'.

It would be useful for periodics to define a wall clock start and offset, for a precise execution time (assuming free workers are available).

Tasks are currently defined like:

    def task(self, func: Optional[Callable]=None, name: Optional[str]=None,
             queue: Optional[str]=None, max_retries: Optional[Number]=None,
             periodicity: Optional[timedelta]=None,
             max_concurrency: Optional[int]=None):
        """Decorator to register a task function.

I propose that we extend the parameters to accept a periodicity_start, another datetime.timedelta object that is used to state the offset from the next largest unit. For example:

juledwar commented 3 months ago

This should be simple to implement: the register_periodic_tasks can change the "now" value to the required starting offset as per the task definition.

The hard part is making sure the parameters make logical sense and rejecting bad ones.

juledwar commented 3 months ago

Additionally, any existing tasks already in Redis that are subsequently changed to include a periodicity_start would need updating in Redis. In practice, this means going over all of them on every worker startup to make sure they are defined correctly.