clokep
commented
5 years ago I was recently working to integrate Celery into a Twisted process we run in a better way (see https://patrick.cloke.us/posts/2018/10/23/calling-celery-from-twisted/ / https://github.com/clokep/twistedcelery). One thing I noticed is that much of the Celery code-base seems to combine three concepts into single method calls (which makes it difficult to have both a sync and an async version that do similar things):
- Message creation (e.g. choosing which queue to send to)
- Message serialization (e.g. converting the message body to a set of bytes)
- Message sending (e.g. here's a dict of headers and a bundle of bytes, please put them into your broker specific format and send them)
There's also something in there about ensuring that that queue exists already and such. The tl;dr is that it is hard to push any async concepts into Celery without copying huge portions of the internals, at least as far as I've been able to see. (Note that this comment is about using async. code as a producer -- sending tasks and looking at results -- not about using it for a worker. I'm not 100% sure what this issue is about.)
This vaguely reminds me of a talk I saw at PyCon a couple years ago: https://us.pycon.org/2016/schedule/presentation/1743/ / https://www.youtube.com/watch?v=7cC3_jGwl_U
auvipy
https://github.com/erdewit/distex this could be a good candidate