chromano / celery-workflows

Useful library for dealing with tasks DAG in Celery
Apache License 2.0
15 stars 5 forks source link

Need to specify you only support restricted DAG - single parents only #1

Open davmlaw opened 6 years ago

davmlaw commented 6 years ago

Hi, your project looks nice, I needed to get method signatures so I ended up modifying it to pass in a dictionary eg:

signatures = {"notify_payment" : notify_payment.si("1")}

and get tasks like:

    def _get_task(name):
        task_name = task_id(name)
        return self.signatures.get(task_name, signature(task_name))

However, I ended up not being able to use it because my nodes can have multiple parents. Maybe you could put a note in there about that, or get that to work. I think you'll need to define the JSON as child : dependency instead. Good luck!

chromano commented 6 years ago

Hi Dave! Thanks for opening this issue, this is a feature I decided not to implement because I wanted to start really small and I was not sure this would be even necessary. My idea was to allow to refer to other nodes as well as to the Celery tasks itself. For example:

    {
        'task0': {
            'task01': {
                'task011': { },
            },
            'task02': {
                'task01': { },
                'tasks021': { }
            }
        }
    }

In this case, task01 has two "parents". However, it seems a little bit unclear to me if putting tasks inside task02.task01 means these tasks will execute at the same time as the tasks defined in task01 or if this should be considered an overwrite of tasks01. I guess the later as this allows more flexibility.