Otel Java agent proxy config

[jpvallez]

Hi Otel community!

Much like https://github.com/open-telemetry/opentelemetry-java/issues/6204, I am trying to export from the otel java agent to an otlp endpoint through a proxy. We are unfortunately unable to use the collector in this instance so we’re looking for a way to configure proxy settings on the java agent.

I note that the feature to enable this has been added in https://github.com/open-telemetry/opentelemetry-java/pull/6270.

I’m not sure if it’s possible but - Can we configure proxy settings for the otlp exporter in the otel java agent by setting for example jvm flags or any other config? Or at this stage is it only possible by interacting with code/writing an extension as mentioned in https://github.com/open-telemetry/opentelemetry-java/issues/6204#issuecomment-2072541892

I also note the feature request here https://github.com/open-telemetry/opentelemetry-specification/issues/4007 which would be the most ideal solution to our problem.

Thanks!

[jkwatson]

moving this to the instrumentation repository, since it's agent-specific

[sxjthefirst]

We have this issue too. Sometimes we can't have a local collector in environments and have to traverse a proxy to get to one. We have two issues:

1. JAVA Agent seems to not be able to connect to HTTP/S_PROXY set in env or with a -D equivalent and throws an error.
2. Ideally we want to use a proxy setting for ONLY the agent so that we don't affect any other connections the monitored java process might be making.

[breedx-splk]

Yeah right now it would need to be done with code, using the SPIs (see the ConfigurableSpanExporterProvider and the config PR mentioned above).

At least before the spec gets exporter proxy settings, I think an agent extension that configures the exporter for proxy would be a worthy addition to opentelemetry-java-contrib, if you're willing to build and maintain such a thing.

[laurit]

The agent can be customized via extensions. Extensions can implement the SPIs from https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure-spi and use them to customize the opentelemetry sdk used by the agent. Unfortunately there doesn't seem to be an SPI that would let you easily configure the proxy settings. I guess the ConfigurableSpanExporterProvider mentioned by @breedx-splk or perhaps https://github.com/open-telemetry/opentelemetry-java/blob/18d192d1fadc6f6dd4dfbff01c0c139eaf93c604/sdk-extensions/autoconfigure-spi/src/main/java/io/opentelemetry/sdk/autoconfigure/spi/AutoConfigurationCustomizer.java#L64 could be used. You can use addSpanExporterCustomizer to replace the auto configured exporter with another one configured by yourself that has the proxy settings. This isn't really ideal as you will need to configure the exporter from scratch. @jkwatson could this issue be moved back to the core repository so that @jack-berg can take a look. In my opinion this isn't as much about the agent but more about how to configure the proxy with autoconfigured sdk and whether something could be done to make such configuration less painful.

[jack-berg]

This isn't really ideal as you will need to configure the exporter from scratch.

You can "enhance" the exporter by calling toBuilder() with something like:

public class ProxyCustomizer implements AutoConfigurationCustomizerProvider {
  @Override
  public void customize(AutoConfigurationCustomizer autoConfiguration) {
    autoConfiguration.addSpanExporterCustomizer((spanExporter, configProperties) -> {
      if (spanExporter instanceof OtlpHttpSpanExporter) {
        spanExporter.shutdown().join(10, TimeUnit.SECONDS);
        return ((OtlpHttpSpanExporter) spanExporter)
            .toBuilder()
            .setProxy(ProxyOptions.create(InetSocketAddress.createUnresolved("host", 8080)))
            .build();
      }
      return spanExporter;
    });
  }
}

In terms of making it less painful, declarative configuration is my answer. I would like to see proxy options be part of the OTLP exporter schema in the declarative configuration data model, which would avoid the need for any customization.

[sxjthefirst]

@jack-berg just to clarify do we need the declarative config update for creating a new otel parameter for the java agent?

I am not sure if that also includes the agent not being able to connect using the JVM level proxy. We don't see an error message but the env vars aren't picked up. We get a connectivity error even if we start the process as below (bug? agent doesn't read proxy?):

export HTTP_PROXY=blah:bleh
export HTTPS_PROXY=blah:bleh
java -Dotel.service.name="testapp" myapp

[trask]

@sxjthefirst can you try instead with system properties here https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html?

[jpvallez]

Hi everyone, thanks for the responses.

@trask - setting proxy with system properties as per https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html seems to work. For those following along, note this is a jvm wide setting. E.g. java -javaagent:opentelemetry-javaagent.jar -Dhttp.proxyHost=proxyhost -Dhttp.proxyPort=port -jar jar-file.jar

This answers my question but still very keen on something exporter specific such as declarative configuration.

Thanks!

[Starefossen]

Same as the author but we set proxy and would like it to exempt the specific host like so where we excempt the the namespace the collector is running in from the proxy list:

  containers:
    - env:
        - name: HTTP_PROXY                                                                                                                 
          value: http://some.proxy.org:8088/                                                                                                 
        - name: http_proxy                                                                                                                 
          value: http://some.proxy.org:8088/                                                                                                 
        - name: HTTPS_PROXY                                                                                                                
          value: http://some.proxy.org:8088/                                                                                                 
        - name: https_proxy                                                                                                                
          value: http://some.proxy.org:8088/                                                                                                 
        - name: NO_PROXY                                                                                                                   
          value: *.some-namespace                                                                                                                       
        - name: no_proxy                                                                                                                   
          value: *.some-namespace
        - name: JAVA_PROXY_OPTIONS
          value: -Dhttp.proxyHost=some.proxy.org -Dhttps.proxyHost=some.proxy.org -Dhttp.proxyPort=8088 -Dhttps.proxyPort=8088 -Dhttp.nonProxyHosts=*.some-namespace

[trask]

@Starefossen can you try setting http.nonProxyHosts (from https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html)?

I'm not certain whether OkHTTP honors this or not, but it's worth a try.

[Starefossen]

@trask -Dhttp.nonProxyHosts=*.some-namespace is set via JAVA_PROXY_OPTIONS and does not apear to be respected by OkHTTP in this instance.

[krismarc]

@trask -Dhttp.nonProxyHosts=*.some-namespace is set via JAVA_PROXY_OPTIONS and does not apear to be respected by OkHTTP in this instance.

While java application has no java proxies configured then it works fine, I am able to connect. After adding java proxy settings, it starts to fail with timeouts.

[otel.javaagent 2024-10-15 12:32:32:245 +0200] [OkHttp https://otel.ingest.***.com/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message: timeout

Therefore, I assume, it understands there's proxy configured like -Dhttp.proxyHost however, it doesn't honor -Dhttp.nonProxyHosts (or.. actual value syntax is misinterpreted? it contains pipes as separator). Please note our endpoint is considered internal, no proxy needed to reach it. While proxy is needed to reach things behind it for other application purposes. So no matter if I tried to whitelist our target within nonProxyHosts it tries to use proxy and we get a timeout.

Best regards, K.M.

[krismarc]

Ok, I know the problem. It depends on handler which reads nonProxyHosts and makes list out of excluded domains patterns. The problem occurs if value of nonProxyHosts is quoted. If my excluded pattern is leading or trailing one, then double quote character becomes part of value and due to that it's not matching.. and let's it think that it should yet go over proxy.

Basically,

-Dhttp.nonProxyHosts="localhost|127.0.0.1|otel*" or -Dhttp.nonProxyHosts="otel.*|localhost|127.0.0.1" won't work while -Dhttp.nonProxyHosts=localhost|127.0.0.1|otel* or -Dhttp.nonProxyHosts="localhost|127.0.0.1|otel*|" are fine

the very last one makes it different list element not containing double quote :)

Actually those situations are mentioned in various softwares. eg https://confluence.atlassian.com/jirakb/jira-server-ignores-http-nonproxyhosts-jvm-option-214863640.html

while Oracle's documentation gives an example with quotes: https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

All http connections will go through the proxy server at webcache.example.com listening on port 80 (we didn't specify any port, so the default one is used).

$ java -Dhttp.proxyHost=webcache.example.com -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL