PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.5k stars 1.64k forks source link

Retry anyio.EndOfStream exceptions in the client #15935

Open EmilRex opened 1 week ago

EmilRex commented 1 week ago

Describe the current behavior

The Prefect client currently does not retry anyio.EndOfStream exceptions. As a result, when a load balancer instance is suddenly terminated, a flow run that could otherwise connect to a healthy load balancer instance may terminate and be left in a Running state according to the server.

Describe the proposed behavior

The Prefect client should retry anyio.EndOfStream exceptions.

Example Use

No response

Additional context

It's currently possible to specify arbitrary http codes to retry via PREFECT_CLIENT_RETRY_EXTRA_CODES but this is not similarly possible for exceptions.

desertaxle commented 1 week ago

@EmilRex do you have an MRE or a stack trace where an anyio.EndOfStream occurred? That would help us track down where we need to enhance the client's error handling.