[Core feature] Support no task execution in a conditional branch

[yubofredwang]

Motivation: Why do you think this is important?

As a Flyte user, I often have tasks that only need to run under certain conditions, and if conditions not satisfied, no task needs to be executed. Currently, the conditional in Flyte does not support skipping one of the branch without any task execution. This means that we will need to use a noop task to skip the branch. This will lead to waste of time during execution.

For example:

@task
def noop():
     ...

conditional("generate_artifact")
.if_(need_to_generate.is_true())
.then(generate_artifact())
.else_(noop())

This makes the code ugly and UI messy. We will need to figure out a way to cleanly handle this.

Goal: What should the final outcome look like, ideally?

Ideally, we can introduce something likeend() or skip() operation in conditional to skip the branch, similar to fail() operation here: https://github.com/flyteorg/flytekit/blob/master/flytekit/core/condition.py#L295

Describe alternatives you've considered

noop() task with cache is being used right now. it is bad coding practice and confusing in the UI.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[ALMerrill]

We'd also be interested in this feature, including being able to get return values from the conditional task, defaulting to a default value. This would clean up the UI a lot, rather than having several steps that just say "branchNode" and having to open all of them to get to the actual task

[dosubot[bot]]

🤖

Hello @yubofredwang! I'm Dosu, a helpful bot here to assist you with your issues and questions about Flyte while we wait for a human maintainer to chime in. Just know that I'm here to make your experience as smooth as possible. Let's dive into your issue!

Thank you for your detailed request. I understand the need for a more elegant solution to handle conditional branches in Flyte where no task needs to be executed.

Currently, as you mentioned, the workaround is to use a noop task. This task is implemented as a No-Operation Resource Manager in the Flyte codebase, which doesn't actually manage any resources but fulfills the interface contract of a resource manager. You can find more details about its implementation here.

As for the conditional execution, it is implemented through the use of events and states. The codebase contains several functions that handle the creation, updating, and retrieval of node executions, which are the individual tasks within a workflow. More details about its implementation can be found here.

Your suggestion to introduce an end() or skip() operation in the conditional to skip the branch is a valid one. However, implementing this feature would require changes to the core execution logic of Flyte. It would involve modifying how node executions are created, updated, and retrieved, as well as how the input and output data of these node executions are managed.

While I can't create a pull request or issue for this feature, I would recommend discussing this further with the Flyte community. They might be able to provide more insights into the feasibility of this feature and the potential challenges in implementing it.

I hope this information is helpful. If you have any other questions or need further clarification, please don't hesitate to ask.

Best regards, Dosu

Sources

• flyteadmin/pkg/manager/impl/node_execution_manager.go
• flytepropeller/pkg/controller/nodes/task/resourcemanager/noop_resourcemanager.go

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kumare3]

I would model this using a Flytepropeller task-plugin, that is as simple as an echo. The default echo could be None?

An example of implementing a task-plugin. this has no overhead besides state transition. Here is an example plugin that simply sleeps for some period of time - https://github.com/flyteorg/flytepropeller/pull/362/files

I would recommend implemeting echo that returns the passed in promise or literal. And this to return a None value we could write

from flytekit import echo

conditional("generate_artifact")
.if_(need_to_generate.is_true())
.then(generate_artifact())
.else_(echo(i=None))

We could also just do this on flytekit side by implementing

none_return = functools.partial(echo, None)

[eapolinario]

Can we take a step back before jumping into that idea of introducing an empty echo task to represent empty else branches?

This solves the immediate problem, but it'll be very confusing in the flyteconsole's graph view.