The Prefect client currently does not retry anyio.EndOfStream exceptions. As a result, when a load balancer instance is suddenly terminated, a flow run that could otherwise connect to a healthy load balancer instance may terminate and be left in a Running state according to the server.
Describe the proposed behavior
The Prefect client should retry anyio.EndOfStream exceptions.
Example Use
No response
Additional context
It's currently possible to specify arbitrary http codes to retry via PREFECT_CLIENT_RETRY_EXTRA_CODES but this is not similarly possible for exceptions.
@EmilRex do you have an MRE or a stack trace where an anyio.EndOfStream occurred? That would help us track down where we need to enhance the client's error handling.
Describe the current behavior
The Prefect client currently does not retry
anyio.EndOfStream
exceptions. As a result, when a load balancer instance is suddenly terminated, a flow run that could otherwise connect to a healthy load balancer instance may terminate and be left in aRunning
state according to the server.Describe the proposed behavior
The Prefect client should retry
anyio.EndOfStream
exceptions.Example Use
No response
Additional context
It's currently possible to specify arbitrary http codes to retry via
PREFECT_CLIENT_RETRY_EXTRA_CODES
but this is not similarly possible for exceptions.