search

girder / girder_worker

Distributed task execution engine with Girder integration, developed by Kitware
http://girder-worker.readthedocs.io/
Apache License 2.0
34 stars 30 forks source link

Running tasks in parallel #51

Open rbaral opened 8 years ago

rbaral commented 8 years ago

Hi,

Does the girder worker support parallel execution of tasks inside a workflow. For instance, lets say I have a workflow having tasks T1,T2,T3,T4... Is there a way I can run tasks T2 and T3 in parallel after task T1 is completed?

Thanks.

jeffbaumes commented 8 years ago

This feature is not implemented as of yet in girder worker directly. To do this you would need to start the girder-worker server and send the various tasks through celery manually (calling run). Celery supports parallelism with groups.

I'd like to keep this issue open to track this if we want to implement a better solution that girder worker handles directly.

rbaral commented 8 years ago

Yes, celery does provide the parallel execution, but I need to run some tasks of the workflow itself to run in parallel. I can run the whole workflow instances in parallel using celery or raw python parallelism. The parallelism within the workflow is the good to have feature and also highlighted in the workflow patterns [http://www.workflowpatterns.com/patterns/control/]

Thanks.

jeffbaumes commented 8 years ago

Agreed, it would be a great feature to have.