flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.79k stars 660 forks source link

[Docs] Enhance performance optimization guide #5745

Open davidmirror-ops opened 2 months ago

davidmirror-ops commented 2 months ago

Description

This docs page was recently updated but some issues persist and should be addressed to avoid users spending time trying to configure parameters that are not located where they should be or trying to understand obscure concepts.

  1. The keys in the Configuration parameter in this table refer to backend plugin workers and not core.propeller settings.
  2. For some users it's still not clear from the explanation in this section if parameters like workflow-reeval-duration or downstream-eval-duration should be increased or decreased to prevent propeller to pick new workflow evaluations more often than checking existing workflow state.

@cpaulik please add more gaps if you find them

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

cpaulik commented 2 months ago

In terms of explanation the streak-length is also quite unclear to me.

I was not able to understand what it actually does from the docs but in terms of just optimising the value.

The current page says

Larger values can lead to faster completion times for workflows that benefit from continuous processing, especially cached or computationally intensive workflows,

and at the same time that lower values are

effectively prioritizing fast-changing or “hot” workflows.

It's unclear to me what workflows that benefit from continuous processing are and how they are different from prioritizing fast-changing or “hot” workflows