search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.17k stars 550 forks source link

[Core feature] Build multiple ImageSpec in parallel #5474

Closed EngHabu closed 2 weeks ago

EngHabu commented 2 weeks ago

Motivation: Why do you think this is important?

Having a complex workflow with multiple tasks that use different ImageSpecs. The current behavior is that it goes sequentially building each image and pushing it to the registry before moving to build the next one.

Goal: What should the final outcome look like, ideally?

Ideally all ImageSpecs can run in parallel (or maybe parallelism can be controlled)

Describe alternatives you've considered

Drink coffee and wait

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

runllm[bot] commented 2 weeks ago

Currently, Flyte does not support building multiple ImageSpec in parallel. The build process for each ImageSpec is executed sequentially. This limitation can lead to longer build times for complex workflows with multiple tasks using different ImageSpecs (source).

Would you like more details on potential workarounds or further information on this topic?

Sources:

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @runllm to ask follow-up questions. See runllm.com for more!

EngHabu commented 2 weeks ago

Dupe https://github.com/flyteorg/flyte/issues/4671