Support other task queues than Celery

[vhtkrk]

Celery is the classical choice for a task queue in Python/Django land, but it is a bit on the heavy side and quite frankly an overkill for many projects that just need a simple task queue without the kitchen sink included.

For that, our project has been using huey with its included huey.contrib.djhuey Django integration. There also exists a django-huey which I've understood is somewhat popular, and adds some more options like multiple task queues on top of what is still mostly huey.contrib.djhuey. We also considered Django Q as an alternative that seemed very polished but I don't have personal experience with that.

Personally I would benefit most from a huey.contrib.djhuey integration, but I like to think the others would be useful for someone else too.

[wilhelmklopp]

@vhtkrk 👋 Thanks for opening this and sorry for my slow response.

We've been thinking about how to best support huey (and in fact any other library code which it might be useful to inspect) and we have a couple of ideas. Nothing to share yet, but will keep you posted on here as we make more progress ✨

One question I have: In local development, do you use huey in such a way that tasks are executed eagerly, so without the need to run a separate local consumer process? (in celery this is what the CELERY_ALWAYS_EAGER setting does)

Asking because this has an impact on how easily Kolo is able to inspect the code that executes within the context of a task

[vhtkrk]

Thanks for the update! 🙂

So far I've been using a separate consumer process, but it's not set in stone and there's no reason I couldn't use immediate mode (that's what Huey calls it iirc) if Kolo were to require that.

[simkimsia]

Adding another vote for huey

I started using huey in greenfield projects as default instead of celery

[LilyFoote]

Support for huey has been added (actually a while ago). Other task queues may be added in the future.