PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.34k stars 1.64k forks source link

Allow adding retryable http status codes #7749

Open hateyouinfinity opened 1 year ago

hateyouinfinity commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

Currently, the list of retryable http status codes is hardcoded.

Describe the proposed behavior

It would be useful to be able to specify additional retryable status codes (e.g. via an environment variable). This set of codes would then be used by the client along with existing status codes (429/503).

Example Use

Sometimes corporate proxies misbehave and throw "407 Proxy Authentication Required". Those issues are ultra transient, but currently they sometimes lead to the flow failing.

Additional context

No response

zanieb commented 1 year ago

👍 this makes sense to me, but we do not want people retrying on error codes like 500 (here, we have to worry about idempotency) — maybe we should have an allow-list or banned-list of codes that can be used in the setting.