New features

[AzHicham]

Hello

First of all thanks for your amazing work !! Here are some features that could be really amazing

• Composition / Pipeline like in Dramatiq, or may be add a way to send a task B within a task A
• Handle sigterm signal and do a warm shutdown eg stop listening for more jobs and wait for active job to finish
• Add priority to message handling
• Is there a way to do some unit testing with mocking or a stub (eg without running a Redis for example)

Few questions btw, does a worker run in a dedicated thread ? Is it possible to run multiple instance of worker app ?

[geofmureithi]

Hello So, some of the issues you have raised will be covered in version 0.4 (#12) specifically pipe-lining and shutting down. On priority handling, It is currently not a major priority, as one can achieve this by running multiple workers or other workarounds. On the issue of testing, there will be improvements in v0.4, but we may also add MockStorage.

On the other questions:

• does a worker run in a dedicated thread ? No, workers are run on a tokio thread pool
• Is it possible to run multiple instance of worker app ? Yes, depending on what you are asking, eg register_with_count for multiple workers, run multiple process (apart from cron), or run multiple machines(apart from sqlite & cron)

[AzHicham]

Hi

Thats really cool for pipe-lining, shutting down and mocking !! For priority I understand there is plenty of ways to do it, with multiple worker for instance !

• Ok, As long as we can limite the number of concurrent worker that's fine !! I tried and it works fine !!
• I did a simple test with 1 client + 2 application worker (eg simulate scaling in kubernetes) and It works like a charm !!

May be one last question, right now in my kubernetes deployement I scale workers according to the number of message in my celery/redis queue. For apalis/redis do you know the name queue I need to watch ?

EDIT: And maybe I found a bug: I started a worker then restarted Redis, The worker app was not able to receive job anymore :/

Thanks a lot !!

[AzHicham]

Hello @geofmureithi Idk if you saw my last edit about a potential bug when Redis restart while worker is already connected :)

[geofmureithi]

Hey, can you create an issue with the steps to replicate this?

[geofmureithi]

Hey @AzHicham Does #54 Fix your issues?

[AzHicham]

Sorry for the late reply I'll try this asap :)