search

closeio / tasktiger

Python task queue using Redis
MIT License
1.42k stars 80 forks source link

Passing "schedule" as a keyword argument to delay() #102

Open mrname opened 6 years ago

mrname commented 6 years ago

Hey team! First of all great work on tasktiger, it rocks.

I have a need to be able to dynamically create tasks with different user defined schedules. As such, I cannot provide a schedule when decorating the task function, and instead need to do it when I call delay on the task. I see in the docs that schedule is not a supported keyword argument, and was thinking about working on a pull request to add support for this, but I assumed that this was an intentional design decision for some reason. Could you provide more information about why this is not available as a kwarg for the delay call?

Thanks!

mrname commented 6 years ago

After spelunking a bit, I see that if a task is scheduled it is considered as unique, where unique indicates that no more than one task of the given method can run at a time (regardless of arguments). As such, it would seem that creating multiple instances of a task with the same function but different schedules would not work anyway, because they cannot execute at the same time.

I'm guessing that (as much as I hate to say it) tasktiger is potentially not a good fit for my current efforts, and I will need to use something more robust (and gross) like celery to get the job done.

If this is not the case, please point me in the right direction, as I would rather issue a PR to tasktiger then use celery.

thomasst commented 6 years ago

Can you clarify on what schedule should do exactly? If you want to execute a task at a certain time you pass when, which doesn't consider the task to be unique. Only periodic tasks are unique.

thomasst commented 6 years ago

I think I see what you want. The way schedule was designed is per task function, and it's not currently meant to scale by queueing many tasks with different args and schedules. Whenever TaskTiger starts, we queue any missing tasks with a schedule for periodic execution. If you have a good way to solve this on a per task/arg basis let me know.

themonk911 commented 6 years ago

+1 this is a feature I would use too. As an example, rq-scheduler (https://github.com/rq/rq-scheduler) allows dynamic creation of periodic/scheduled tasks for python-rq from version 0.3 onwards. The scheduler runs as a separate process that polls redis every minute for new scheduled tasks, and adds them onto the relevant queues.