Forward `HTTP_PROXY/HTTPS_PROXY/NO_PROXY` to components.

[AndersonQ]

Describe the enhancement:

Currently there are 3 different proxies that can be configured for the Elastic Agent:

• proxy to communicate with ES
• proxy to communicate with Fleet Server
• a default proxy.

The default proxy is set though the environment variables HTTP_PROXY/HTTPS_PROXY/NO_PROXY. This is actually a Go feature, we don't handle them explicitly, but any other proxy will override it.

The proxy setting doe thorough HTTP_PROXY/HTTPS_PROXY/NO_PROXY are inherited by the components the Elastic Agent runs as sub process, that means the beats "receive" those settings when they're set for the Elastic Agent, but other components don't. What is at least an inconsistent behaviour.

Describe a specific use case for the enhancement or feature:

A user using the environment variables to set up a global proxy and having problems because some of the Elastic Agent components, such as Endpoint Security, aren't receiving the proxy.

[mbudge]

Wouldn’t it be better if this was ELASTIC_HTTPS_PROXY so we can manage the proxy when elastic agent is deployed, without other proxies overriding the setting.

The proxy env variables could be created by elastic agent through command line parameters.

https://github.com/elastic/elastic-stack-installers/pull/212#issuecomment-1879514732

[cmacknz]

Setting HTTPS_PROXY or HTTP_PROXY will set the proxy for every connection made by Elastic Agent, except those that originate from Elastic Defend which runs as an independent protected service and so doesn't inherit this environment variable. Correcting this inconsistency where the _PROXY environment variables don't affect Elastic Defend is what this issue is primarily about.

You can also control the proxy for each individual connection (Fleet, Elasticsearch or Output, and Upgrade downloads) independently directly in the agent policy.

Unless I'm missing something here, I don't know why we'd introduce another proxy environment variable.

[mbudge]

Ignore me the proxy stuff with fleet worked fine.

[amitkanfer]

@cmacknz @AndersonQ what's the action item here?

[cmacknz]

It would be forward the value of the *_PROXY environment variables set on the elastic-agent process to the endpoint-security process using the control protocol, so that users don't have to set them in two places.

[pierrehilbert]

@cmacknz do you have an estimate for this one? I would like to plan it in our next coming sprint.

[intxgo]

It would be forward the value of the *_PROXY environment variables set on the elastic-agent process to the endpoint-security process using the control protocol, so that users don't have to set them in two places.

Yes, it was already discussed, but some technical dependencies were holding us back then at Endpoint side. We're already ready to proceed with it. It'll be quite straightforward to implement, the foundation is already there in Elastic Defend sources.

[cmacknz]

We need to confirm and document the precedence order of the proxy configuration fields as part of this. That is if endpoint gets the contents of the HTTPS_PROXY environment variable and also the proxy_url key is set on the output, which one does it prefer?

[amitkanfer]

@cmacknz can / should we compile the effective proxy in fleet / agent and pass only that to endpoint?

[jlind23]

We need to confirm and document the precedence order of the proxy configuration fields as part of this. That is if endpoint gets the contents of the HTTPS_PROXY environment variable and also the proxy_url key is set on the output, which one does it prefer?

I guess they shouldn't look up for environment variables when it comes to proxy settings and only rely on values sent down though the control protocol. @ferullo this is what we discussed about yesterday right?

[ferullo]

Yeah, it makes sense to me that Endpoint won't consider environment variables at all once this is implemented, but I'll defer to @intxgo and @nfritts on that.

[cmacknz]

I guess they shouldn't look up for environment variables when it comes to proxy settings and only rely on values sent down though the control protocol.

That is the expectation yes. For the variable precedence, we need to test and document this from the agent perspective so that it knows what it should be sending to endpoint.

The HTTPS_PROXY and related variables are features of Go, we do not test them and only document that they exist. There is no documentation explaining how it interacts with the individual proxy_url variables yet.

@cmacknz can / should we compile the effective proxy in fleet / agent and pass only that to endpoint?

We could, I like the simplicity of this as from endpoint's perspective there are no code changes if they already respect the individual proxy_url variables. As mentioned earlier we just need to make sure we use the right precedence rules.

The only thing I don't like about sending the effective proxy is that the output and fleet configuration sent to endpoint no longer matches what is in the agent policy. One way to solve this would be to keep proxy_url as the effective proxy, but also include the other values used to compute it with new keys that are informational only. For example we could put the value of HTTPS_PROXY in an env_https_proxy_url and the one from the policy as policy_proxy_url or something so that looking at the endpoint output unit still has all the information needed to debug proxy behavior.

[intxgo]

regarding the precedence rules, we've already implemented that on Endpoint side. Native proxy setting from policy is taken first, if it's not set, when we fallback from proxy from agent (which we don't have yet).

[cmacknz]

Doing a quick test locally, on the side agent it looks like if I set both HTTPS_PROXY or HTTP_PROXY and the proxy_url for a given connection in the policy at the same time, then the proxy_url in the policy takes precedence.

[intxgo]

Sounds great, IMO as a user I would expect Agent/Endpoint to use what's configured explicitly in the policy rather than preferring some "random" system settings

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@michel-laterman @ycombinator this issue status has been changed to implementation but there is no PR attached to it, is this really started?

[michel-laterman]

There's no pr yet

[michel-laterman]

@cmacknz, I have a draft for the change and @blakerouse rouse some concerns about the intended scope of the change. Should we inject the proxy_url value into every output that does not have it, or do we want to limit the scope to endpoint or endpoint+elasticsearch?

[cmacknz]

Only providing this to endpoint is fine as a starting point, they are the only component that will actually use it today.