Passing a reference to a task as action does nothing

[ksamuel]

When an action list contains a reference to another task, it seems silently skipped.

Environment

1. OS: Ubuntu 20.04
2. python version: 3.9.13
3. doit version: 0.36.0

With this dodo.py:

def task_1():
    return {"actions": ["echo b"], "verbosity": 2}

def task_2():
    return {
        "actions": [
            "echo a",
            task_1,
            "echo c",
        ],
        "verbosity": 2,
    }

We get:

doit 1
.  1
b

As expected.

However, we also get:

doit 2
.  2
a
c

In task_2, the reference to task_1 seems silently ignored.

Either it is allowed to pass references to other tasks as actions, and then doit should run them. In that case, I expect to get:

doit 2
.  2
a
b
c

Or it is not and doit should raise an exception.

However, if it's not allowed, then a mechanism to reference another task in actions should be offered and documented. I looked for it in the doc, but the only way I could find would have been the very verbose:

def task_3():

    yield {
        "name": "task_3_step1",
        "actions": [
            "echo a",
        ],
        "verbosity": 2,
    }

    yield {**task_1(), "name": "task1_in_task3"}

    yield {
        "name": "task_3_step2",
        "actions": [
            "echo c",
        ],
        "verbosity": 2,
    }

    return

Since making an alias for a group of task to run together (but not dependent on each others) is a rather common operation, it should have a concise way to express it.

[schettino72]

"reference to a task"? Sorry, this doesn't exist. where you get this idea from?

[ksamuel]

(Hi @schettino72, thanks a lot for coding doit, I use it on all my projects. I just realized I haven't donated to this tool yet, so I just did :heart:)

So, if passing a task-creator reference in an action list doesn't exist, it should result in an error. Right now it just skips the function silently, which lead me to believe it was correct, but that I had one detail wrong.

However, if it's not possible, then the docs should include a section to explain how to do something like this:

• define task 1 as: run command echo b
• define task 2 as:
  • run command echo a
  • run task 1
  • run command echo c

You can't use a task_deps for this since the task is in the middle, and apart from yield (which as you can see in the example is very verbose), I can't find any way to do it in the docs.

[schettino72]

So, if passing a task-creator reference in an action list doesn't exist, it should result in an error.

A task-creator is just a plain python function. Plain python functions can also be actions... so there is no easy way to detect that. The function is not being skipped, it is being executed.

However, if it's not possible, then the docs should include a section to explain how to do something like this:

The docs have to balance being succinct while explaining the basic concepts. I get way more reports from people who did NOT read the docs than about the docs being incomplete. Said that, there is a plan (for several years) to add a section with more hands on docs.

    define task 1 as:
    run command echo b
    define task 2 as:
        run command echo a
        run task 1
        run command echo c

It is hard to give advice when I dont know the specific details of what the tasks or commands are doing.

• task_dep is one way of doing it. Tasks needs to be composed on DAG. You can not just call it in the middle of another task. How would the DAG look like?

• tasks also provide a "setup" and "teardown" feature, maybe it applies to your use-case.

(which as you can see in the example is very verbose), I can't find any way to do it in the docs.

Trade-offs, when you have to build a DAG, it is more verbose than just calling functions. No way around that. Since tasks are plain python dict, you could use some python code to automatically chain tasks, but doit tries to provide a minimum interface and allow users to customize the task creation in a way that is efficient for their domain.

[ksamuel]

A task-creator is just a plain python function. Plain python functions can also be actions... so there is no easy way to detect that. The function is not being skipped, it is being executed.

Right, my mistake, indeed.

Said that, there is a plan (for several years) to add a section with more hands on docs.

I have several FOSS projects with docs lagging being myself, so I get it :)

You can not just call it in the middle of another task. How would the DAG look like?

You would have 2 DAG.

A inefficient way of picturing it would be:

def task_foo():
    ...
def tast_bar():
     ...
    "actions": ["echo a", "doit foo", "echo 2"]

Of course, this has the side effect of creating a whole new process, which is overkill. But it's a good way to see how it could work.

doit could do that in a lighter way, without the subprocess. Let's imagine something like this:

...
from doit import TaskAction # hypothetical function that creates an explicit reference to a task

"actions": ["echo a", TaskAction('foo'), "echo 2"]
...

TaskAction('foo'), in this context, would have a reference to the task generated from task_foo. When encountering it, doit would create a second DAG for it, independent of the one from task bar. The newly created DAG cannot contain task_bar in it, otherwise it raises an error, to avoid circular dependencies. Then it run the tasks from this new DAG as usual, but separately from the DAG (and task list) from task bar. If it succeed, the it deletes the temporary DAG and task list from task foo, and goes back to the DAG of task bar, carrying on with the execution.

Of course, discussing with you, I realize why it doesn't exist, and how much work it would be. I was naive to think it was a simple matter.

However, since it's something I've been fighting with, I can't be the only one. If the feature is overkill to write, then updating the doc to stipulate you cannot include a task in the middle of actions would at least let people know it's a dead end.

If however, the feature has value to you, I can open a ticket with just the feature request. 

[schettino72]

I guess that would push complexity in wrong place. You can build a helper to define the DAG in a more friendly way to your use-case..