flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.78k stars 659 forks source link

[Core feature] Support specifying workflow labels for the default launch plan in the workflow definition #5774

Closed swarup-stripe closed 1 month ago

swarup-stripe commented 1 month ago

Motivation: Why do you think this is important?

Currently workflow labels/annotations can only be supplied when you launch an execution or when you create a launch plan that's not the default. This creates a big friction point to specifying labels since you have to manually toggle something in the Flyte Console (either to specify the label or switch the launch plan to the one with the label). This will help us add workflow-level metadata and propagate that to all tasks easier.

Goal: What should the final outcome look like, ideally?

Specifying labels directly in the workflow definition like the decorator e.g. @workflow(labels=Labels({"sample-label": "sample-value"}).

Describe alternatives you've considered

We can currently specify it when creating a launch plan but this involves manually switching the launch plan in Flyte Console which is prone to accidents. We can also directly enter in the Flyte Console when launching executions. But again this is prone to accidents. Since the label metadata is semantically tied to the whole workflow not just that particular launch plan, this isn't ideal. We can also leverage pod_template to specify labels to each pod but we'd have to annotate each task in the workflow manually.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

davidmirror-ops commented 1 month ago

Original post on Slack: https://flyte-org.slack.com/archives/CP2HDHKE1/p1726850294050909

Mecoli1219 commented 1 month ago

take

pingsutw commented 1 month ago

@swarup-stripe, if we support setting another launch plan as default, would that address your issue? Therefore, you won't need to switch the launch plan in Flyte Console.

you can create a new LP with your labels, and set it as default.

swarup-stripe commented 1 month ago

Thank you all for the quick turnaround time and for the PR @Mecoli1219!!!

@pingsutw That could technically address it, but ideally we want to enforce having a label and avoid having any unlabelled launch plans so this is definitely the better solution.