OpenTelemetry HTTP Error 404 sending Otel signals to Elastic using OTLP/HTTP

[tobiasrodestock]

Jenkins and plugins versions report

Environment

```text Paste the output here ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Machine operating system: Linux Jenkins version: 2.303.3.3 OpenTelemetry plugin: 1.2.1 Elastic APM Server: 7.15.1

Reproduction steps

1. Setup OpenTelemetry Plugin
2. Configure Elastic APM Server Url e.g. http://url:8200
3. Use OTLP/HTTP

Expected Results

The plugin sends the metrics and traces to the Elastic APM Server.

Actual Results

Feb 17, 2022 1:28:42 PM WARNING io.opentelemetry.sdk.internal.ThrottlingLogger doLog Failed to export spans. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found

Anything else?

The plugin sends the metrics to url:8200/v1/metrics and the traces to url:8200/v1/traces but the Elastic APM Server only listens on url:8200/intake/v2/events.

[v1v]

Hi @tobiasrodestock

Thanks for your issue and details, I tried to reproduce the issue in a local environment with a kind of similar versions (bear with me 2.303.3.3 is not a version I can use but 2.303.3 instead ) and I was not able to reproduce the error.

What did I try?

git clone https://github.com/v1v/opentelemetry-plugin
cd opentelemetry-plugin
git checkout feature/apm-server-demo
make -C demos build
make -C demos start

Then, go to http://localhost:8080/ and click on the Hello pipeline and build.

And you will see traces in Kibana (http://localhost:5601):

Can you confirm if the above works in your end? Just to be sure we can discard if the issue is regarding the plugin itself of the backend configuration?

[tobiasrodestock]

Hey, thank you for taking your time. It did work locally with the things you suggested.  I did manage to make it work by using the OTEL Collector and Exporter. To me, it seems as the Elastic APM Server only supports gRCP and not HTTP. The corporate proxy that is between the Jenkins machine and the machine where Elastic is running is probably responsible for gRCP not working. When I now send the metrics via HTTP to the OTEL Collector that is running on the same machine as the APM Server and then export them with the OTEL Exporter via gRCP it is working just fine.

[cyrille-leclerc]

To me, it seems as the Elastic APM Server only supports gRCP and not HTTP.

Good catch @tobiasrodestock . We are improving our documentation https://github.com/elastic/apm-server/pull/7505

[cyrille-leclerc]

Fixed with https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html#open-telemetry-otlp-limitations

[cyrille-leclerc]

Hello @tobiasrodestock , we run a survey to better understand Jenkins Observability needs and more broadly CI/CD Observability needs, could you by any chance answer it? https://docs.google.com/forms/d/e/1FAIpQLSezbndR1bpPj4YeWquLLWEYOkEjEI-lSyNbtDUVrR-udpt2lA/viewform?usp=sf_link

[v1v]

For the record, https://github.com/jenkinsci/opentelemetry-plugin#using-the-opentelemetry-otlphttp-rather-than-otlpgrpc-protocol solved the issue in our case.