Slow task execution and lack of fault tolerance for server connection failures

zsio commented 4 months ago

Describe the current behavior

Task Execution Speed: Currently, even simple tasks like a single print statement take more than a second to execute. This occurs even for basic operations that should be near-instantaneous.
Server Connection Dependency: When using Prefect server for monitoring task execution, if the network connection to the server fails, it affects the execution of tasks and flows on the client-side. The client-side execution is interrupted and shows a connection error, even though we only intend to use the server for monitoring purposes.

Describe the proposed behavior

Optimized Task Execution: Implement optimizations to reduce the execution time for simple tasks. For instance, a task with a single print statement should complete in milliseconds rather than seconds.
Fault Tolerance for Server Connection: Introduce a fault-tolerant mode where tasks and flows continue to run locally even if the server connection fails. This could involve: a) An option to configure the client to continue local execution despite server connection issues. b) A local caching mechanism to store execution data temporarily when the server is unreachable. c) An asynchronous reconnection attempt that doesn't block task execution.

Example Use

from prefect import task, flow, get_run_logger

@task(name="Say Hello", task_run_name="task run name")
def say_hello():
    logger = get_run_logger()
    logger.info("Hello, world----------!")
    print("hi")

@flow(name="My First Flow", flow_run_name="flow run name")
def flow():
    s = say_hello.submit()
    s.result()
    # or 
    say_hello()

flow()

This task takes over a second to complete

Additional context

No response

zzstoatzz commented 4 months ago

hi @zsio - thanks for the issue!

can you please share the output of prefect version as requested by the issue template? prefect 2.x and 3.x have different engines.

a lot has been done between 2.x and 3.x to improve performance, and we've still been working on making running tasks faster (example), as well as minimizing constant back and forth between clients and the server.

For instance, a task with a single print statement should complete in milliseconds rather than seconds

that said, in order to provide the functionality we offer, we will have some performance overhead. Happy to hear suggestions on specific places you think we may benefit from improvements

zzstoatzz commented 4 months ago

@zsio to expand on that example I linked above, if you install prefect>=3.0.0rc13 (or main) you can try out client-side orchestration which is significantly faster than previous versions.

prefect config set PREFECT_EXPERIMENTAL_ENABLE_CLIENT_SIDE_TASK_ORCHESTRATION=True

a gist example

zsio commented 3 months ago

@zzstoatzz I have followed your suggestions and set the environment variable as instructed. While the running speed has indeed improved, I have encountered a new issue. Now, the task running flowchart is no longer displayed under the flow, and instead, it prompts: "This flow run did not generate any task or subflow runs." I'm hoping you can provide some guidance or a solution to address this problem.


@task
def say_hello2(name):
    return f"Hello {name}!"

@flow()
def hello_flow():
    one = say_hello2.submit("one")
    results = one.result()    

hello_flow()

lucasdepetrisd commented 2 months ago

@zsio have you found any solution to this? I didn't enable that setting but after updating to the 3.0.0 release it shows the same "This flow run did not generate any task or subflow runs"

zsio commented 2 months ago

@zsio have you found any solution to this? I didn't enable that setting but after updating to the 3.0.0 release it shows the same "This flow run did not generate any task or subflow runs"您@zsio找到任何解决方案吗？我没有启用该设置，但在更新到 3.0.0 版本后，它显示相同的“此流程运行未生成任何任务或子流程运行”

@lucasdepetrisd No, I am currently considering removing prefect.

PrefectHQ / prefect