Koed00 / django-q

A multiprocessing distributed task queue for Django
https://django-q.readthedocs.org
MIT License
1.83k stars 285 forks source link

Short delayed schedule objects #568

Open jonasN5 opened 3 years ago

jonasN5 commented 3 years ago

I'm looking to delay some tasks for a short moment, between 5 and 10 seconds. I use Redis. I found this in the code:

    def guard(self):
            ...
            counter += cycle
            if counter >= 30 and Conf.SCHEDULER:
                counter = 0
                scheduler(broker=self.broker)

This is too unreliable for my usecase, but if I use a Task, it will be executed immediately. And I guess using a time.sleep is horrible for performance. Is there a way to achieve the short lived Schedule objects?

I use the delayed tasks in a chat application, to send a notification if the user isn't connected. I need the short delay to give clients time to send a response back to the server.

Koed00 commented 3 years ago

I don't think there is a good way to do this reliably with how Django-Q handles schedules currently. Even if we could get down to 5 to 10 seconds, there is still the length of the queue and the current workload to content with. You can probably never achieve this kind of resolution.

You're best bet is to use a broker that support delay queues - like AWS SQS. You can set a 5 second delay for the queue and directly send the task there instead of scheduling it. The queue itself will delay any message before it's send to the consumers.

If you still need a non-delayed queue, you can set up multiple brokers and then specify the broker when you send the tasks.

jonasN5 commented 3 years ago

Thanks a lot for your reply. For policy reasons, I can't use AWS solutions for my project. Could you imagine a work around to achieve this? Any lead would help :) Would some ugly solution like creating a deamon Thread in a Task (not Schedule), which would use time.sleep, do the trick? I don't really care about the result, as long as it works 99.9% of the time. Otherwise I don't any other solution than migrating to Celery, which I would hate, since I like this package.