PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.4k stars 1.59k forks source link

Support HTTPX client proxy for only prefect domains #9917

Open gabcoyne opened 1 year ago

gabcoyne commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

Currently the prefect client supports using NO_PROXY and HTTPS_PROXY to configure proxy settings. This is sufficient for many cases but does not consider how many of our users with proxies are using Prefect. As Prefect runs in customer environments it is not uncommon to take advantage of network locality and interact with resources in the same VPC. In these situations it is relevant to acknowledge that Prefect's API is public and as such requires a proxy to access. Proxy configuration using environment variables is all or nothing, and using no_proxy to allow proxy bypass for all domains except prefect is challenging and depending on environment, unsupported.

Describe the proposed behavior

Provide configuration options at the httpx client instatiated by Prefect to only Proxy prefect API calsl.

Example Use

PREFECT_PROXY = proxy.domain.example

Additional context

No response

jawnsy commented 1 year ago

I think this makes sense. We'll need to consider how this interacts with users relying on the existing behavior, which is the httpx default behavior.

If someone specifies PREFECT_PROXY, I think we should use this proxy when creating our clients (ignoring HTTPS_PROXY, HTTP_PROXY, and NOPROXY variables). Those variables can then be used in other clients running in the same Python context.

If someone specifies HTTPS_PROXY, HTTP_PROXY, or NOPROXY; and does not specify PREFECT_PROXY, then we should continue to support the current behavior, so that we avoid compatibility issues