Task flows and AWS SQS support

[fridex]

Is it possible to use dramatiq to model task flows - run tasks that have dependencies between them? Compared to Celery which has primitives I didn't find anything about this capability in docs (considering this is not possible) - also Celery is quite limited with some advanced task flows. I'm quite interesting in this as we were dealing with this for quite some time so finally I decided to implement selinon.

Also, is there support for AWS SQS?

[Bogdanp]

Is it possible to use dramatiq to model task flows - run tasks that have dependencies between them? Compared to Celery which has primitives I didn't find anything about this capability in docs.

Not out of the box, though this should be possible by leveraging middleware. I want to avoid adding these to the core project unless there is very high demand so I think, for now, an external package would be the solution here.

also Celery is quite limited with some advanced task flows.

They are also quite buggy depending on the storage mechanism you use. Dropped chords are the worst.

Also, is there support for AWS SQS?

There isn't, but it should be fairly easy to write a Broker for SQS. I'm wary of adding more brokers to the core package, however, since that is likely to increase my maintenance burden and the brokers that I care about are already in there.

Selinon looks really cool, btw! :D

[fridex]

Is it possible to use dramatiq to model task flows - run tasks that have dependencies between them? Compared to Celery which has primitives I didn't find anything about this capability in docs.

Not out of the box, though this should be possible by leveraging middleware. I want to avoid adding these to the core project unless there is very high demand so I think, for now, an external package would be the solution here.

Selinon uses Celery just for publishing messages (kombu) and for retrieving task statuses from Celery's result backend so other tasks can be scheduled based on task success/failure. I'm interested in dramatiq as it seems to be more lightweight. But I guess there is no such ecosystem as in case of Celery (things like flower). Also no track of task statuses?

also Celery is quite limited with some advanced task flows.

They are also quite buggy depending on the storage mechanism you use. Dropped chords are the worst.

Mh... never heard of that. Could you provide me more info/point me to some issue?

Also, is there support for AWS SQS?

There isn't, but it should be fairly easy to write a Broker for SQS. I'm wary of adding more brokers to the core package, however, since that is likely to increase my maintenance burden and the brokers that I care about are already in there.

What about directly using kombu in this case?

Selinon looks really cool, btw! :D

Glad to see that ;-)

[Bogdanp]

Also no track of task statuses?

Result tracking isn't a part of Dramatiq, but it should be fairly easy to add. You can find an example of something like that here. This is another thing I would like to keep out of the core unless there is high demand.

Mh... never heard of that. Could you provide me more info/point me to some issue?

As far as I'm aware these are understood trade-offs rather than "issues" but here are a couple cases I can think of off the top of my head:

• When the result backend doesn't support atomic increment. For these types of backends, celery polls to get tasks' results for all tasks in the flow and after a while it just gives up.
• If you're using the memcached result backend and there's a lot of memory pressure so the chord synchronization key gets dropped.

What about directly using kombu in this case?

One of my design goals for dramatiq is for it to be completely (and quickly!) understandable by reading the source. I think using kombu would hurt that understanding.

[fridex]

Thanks for responses! ;)