search

open-telemetry / opentelemetry-java-contrib

https://opentelemetry.io
Apache License 2.0
145 stars 118 forks source link

JMX Metrics: Protocol Error with otlp exporter #351

Open seism0saurus opened 2 years ago

seism0saurus commented 2 years ago

Description I have a running tomcat server with a war file in it. The JMX port 9000 is open without authentication or TLS (only for testing). There are many nice metrics in the console, if I do not define the exporter as otlp. So I assume the JMX connectivity works correctly.

But when I use the otlp exporter to send the metrics to my Elastic APM server, I get the following error:

May 23, 2022 10:32:59 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export metrics. The request could not be executed. Full error message: stream was reset: PROTOCOL_ERROR
May 23, 2022 10:33:04 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.classes.loaded has recorded multiple values for the same attributes.

I use the following command: java -jar opentelemetry-jmx-metrics.jar -config session.properties and the properties from below.

Steps to reproduce At the moment it is an internal application and I have an on-premise Elasticsearch cluster. I will try to figure out how to give you a minimal working example.

Expectation The metrics should be exportet to Elastic APM via the OpenTelemetry exporter.

What applicable config did you use? My session.properties:

otel.jmx.service.url = service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi
otel.jmx.target.system = jvm,tomcat
otel.jmx.interval.milliseconds = 5000

otel.metrics.exporter = otlp
otel.exporter.otlp.endpoint = https://MyElasticServer:8200
otel.exporter.otlp.headers = "Authorization=ApiKey MySecretKey"
otel.exporter.otlp.attributes = service.name=testapp,service.version=0.1,deployment.environment=dev

Relevant Environment Information Version: 1.14.0 OS: Debian bullseye inside WSL2 on Windows10 Java: openjdk 11.0.15

anuraaga commented 2 years ago

Hi @seism0saurus - it looks like you are specifying https:// but no certificate. Unless you have installed the server's certificate into your actual operating system (unlikely), this won't work.

https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html#instrument-apps-otel

You would need to set OTEL_EXPORTER_OTLP_CERTIFICATE to the certificate of the server. Alternatively you may try just http:// to see if that server is exposing the endpoint via plain text or not, though given the Elastic's docs I suspect that isn't the case.