runllm[bot]
commented
1 month ago Eager workflows in Flyte currently always invoke the latest version of tasks or workflows, which can lead to issues with reproducibility and conflicts during development. According to the Flyte documentation, eager workflows execute the latest version of tasks and workflows registered under the same project and domain.
To address this, you propose adding a version field to the eager workflow decorator to pin specific pipeline versions. This would ensure that eager workflows invoke the same version as the currently executing task, potentially by using the FLYTE_INTERNAL_VERSION environment variable.
Would you like more detail on how to implement this or any other specific aspect?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @runllm to ask follow-up questions. See runllm.com for more!
Describe the bug
Currently, eager workflows always invoke the latest version of whatever task or workflow is being executed. This presents a number of problems:
developmentstage, their eager workflows might invoke tasks or workflows from other developersFor our use case, pipeline version pinning is important and this prevent us from taking full advantage of eager workflows.
Expected behavior
Eager workflows should accept a
versionfield in their decorator to allow users to pin specific pipeline versions. If noversionis provided, the eager workflow should default to invoking the same pipeline version as the currently executing task. This could be accomplished by examining theFLYTE_INTERNAL_VERSIONenvironment variable.Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?