pingsutw commented 3 months ago

Motivation: Why do you think this is important?

It's extremely useful to see the metrics, data, logs, and links on the Flyte Deck while the task is running. However, currently, the Flyte Deck HTML is uploaded to the remote bucket only after the task succeeds.

Goal: What should the final outcome look like, ideally?

FlytePropeller

[ ] Propeller should send the deck URI (in NodeExecutionEvent) back to flyteadmin when the task starts running. https://github.com/flyteorg/flyte/blob/5b0d78715c31824f0af156c92fa4fe8971a412a5/flytepropeller/pkg/controller/nodes/task/handler.go#L514

Flytekit

[ ] Upload a dummy deck when the task starts running.
[ ] Add publish_decks API, so users can decide when to upload the decks.
[ ] Upload flyte deck if the task fails.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Flytekit Status</title>
    <style>
        body {
            display: flex;
            justify-content: center;
            align-items: center;
            height: 100vh;
            margin: 0;
            font-family: Arial, sans-serif;
            background-color: #f0f0f0;
        }
        .message {
            background-color: #ffffff;
            padding: 20px;
            border: 1px solid #cccccc;
            border-radius: 5px;
            box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
            text-align: center;
        }
    </style>
</head>
<body>
    <div class="message">
        <p> Flyte decks have not been created yet. </p>
    </div>
</body>
</html>

@task(enable_deck=True)
def foo():
    ctx = flytekit.current_context()
    ctx.publish_decks()

    for epoch in range(100):
       ...
       flytekit.Deck("metrics", ...)
       ctx.publish_decks()

FlyteConsole

[ ] Show the Flyte deck button while the task is running
[ ] Auto refresh flyte deck. Ifame should read the latest deck html in the remote bucket.

https://github.com/flyteorg/flyteconsole/blob/7636e32d2b1dbefb7d08fb79e5ed063c3ad44ca4/packages/oss-console/src/components/Executions/ExecutionDetails/ExecutionDetailsActions/FlyteDeckButton.tsx#L61

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

NA

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

runllm[bot] commented 3 months ago

Currently, Flyte does not support real-time updates to Flyte Decks while a task is running. Decks are only uploaded to blob storage after the task completes. This limitation is discussed in this GitHub issue and this community discussion.

Would you like more detail on the current limitations and potential workarounds?

Sources:

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @runllm to ask follow-up questions. See runllm.com for more!

MortalHappiness commented 3 months ago

take

Future-Outlier commented 2 months ago

Suggestions by @eapolinario

hard parts:

error handling (avoid crashing when rate limit happens)

flyteorg / flyte

[Core feature] Real-time Flyte Deck #5574