OpenTelemetry Auto Instrumentation failing with OpenTelemetry Java Agent 2.1.0

Abhimanyu9988 commented 9 months ago

Describe the bug

OpenTelemetry Java Agent 2.1.0 fails with ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans. The request could not be executed.

Steps to reproduce

To reproduce,

I used Tomcat 11.0 sample application as provided here: https://tomcat.apache.org/tomcat-11.0-doc/appdev/sample/ Just an FYI, Issue exists for Tomcat 8.0 as well.

Made my Dockerfile FROM tomcat:11.0 RUN apt-get update -y && apt-get install -y wget RUN apt-get install -y curl CMD ["bash"] WORKDIR /usr/local/tomcat/webapps RUN curl -O -L https://github.com/Abhimanyu9988/samplejavaapp/raw/main/sampletest.war CMD ["/usr/local/tomcat/bin/catalina.sh", "run"]

I have created a yaml file for Instrumentation `apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: auto-instrumentation namespace: appd-cloud-apps spec: exporter: endpoint: "http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317" propagators:

tracecontext
baggage sampler: type: parentbased_traceidratio argument: "1" env:
name: OTEL_EXPORTER_OTLP_INSECURE value: "true"
name: OTEL_LOG_LEVEL value: "debug"
name: OTEL_TRACES_EXPORTER value: "otlp,logging" java: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.0.0 env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317"
- name: OTEL_JAVAAGENT_DEBUG value: "true"`

My Application yaml :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-app-appdcloud
  labels:
    app: tomcat-app-appdcloud
  namespace: appd-cloud-apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-app-appdcloud
  template:
    metadata:
      labels:
        app: tomcat-app-appdcloud
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: tomcat-app
        image: docker.io/abhimanyubajaj98/tomcat-app-buildx
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-app-appdcloud
  labels:
    app: tomcat-app-appdcloud
  namespace: appd-cloud-apps
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: tomcat-app-appdcloud

Expected behavior

As expected with OpenTelemetry Java Agent 1.32. and prior versions, Version 1.21.0 should work. Just an FYI, Issue persists in 1.20.0 as well

Actual behavior

For 2.1.0 I can see below in my Logs: BatchSpanProcessor{spanExporter=OtlpHttpSpanExporter{exporterName=otlp, type=span, endpoint=http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317/v1/traces, timeoutNanos=10000000000, compressorEncoding=null, connectTimeoutNanos=10000000000, exportAsJson=false [otel.javaagent 2024-02-20 21:29:33:419 +0000] [OkHttp http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans. The request could not be executed. Full error message: Connection reset

For 1.3.2 BatchSpanProcessor{spanExporter=OtlpGrpcSpanExporter{exporterName=otlp, type=span, endpoint=http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317, endpointPath=/opentelemetry.proto.collector.trace.v1.TraceService/Export, timeoutNanos=10000000000, compressionEnabled=false

logs-1.32.0.txt logs-2.1.0.txt

Javaagent or library instrumentation version

OpenTelemetry Java Agent 2.1.0

Environment

JDK: OS:

Additional context

My Collector remained the same

kubectl -n cco get svc
NAME                                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
appdynamics-cloud-operator-metrics-service                        ClusterIP   10.100.49.189    <none>        8443/TCP                                12d
appdynamics-collectors-ds-appdynamics-otel-collector              ClusterIP   10.100.174.207   <none>        4317/TCP,4318/TCP,14317/TCP,14318/TCP   8d
appdynamics-collectors-ds-appdynamics-otel-collector-headless     ClusterIP   None             <none>        4317/TCP,4318/TCP,14317/TCP,14318/TCP   8d
appdynamics-collectors-ds-appdynamics-otel-collector-monitoring   ClusterIP   10.100.201.198   <none>        8888/TCP                                8d
appdynamics-operators-opentelemetry-operator                      ClusterIP   10.100.44.46     <none>        8443/TCP,8080/TCP                       12d
appdynamics-operators-opentelemetry-operator-webhook              ClusterIP   10.100.91.46     <none>        443/TCP                                 12d
appdynamics-otel-collector-service                                ClusterIP   10.100.167.126   <none>        4318/TCP,4317/TCP,14317/TCP,24317/TCP   8d

breedx-splk commented 9 months ago

Hey @Abhimanyu9988 . You have your exporter endpoint set to http://appdynamics-otel-collector-service.cco.svc.cluster.local:4317. Port 4317 is the gRPC port. With the 2.x agent releases, we have changed the default exporter to http+protobuf instead of gRPC.

If your collector is configured to receive http, then you should just be able to change the port to 4318 and it'll work. If not, you'll need to set the protocol back to gRPC. See this.

Abhimanyu9988 commented 9 months ago

You Sir are amazing!! Thank you Breedx-splk

The See this like is broken. And we need to update : https://opentelemetry.io/docs/kubernetes/operator/automatic/ We still have endpoint: http://demo-collector:4317

I will inform AppDynamics team about the same. If you can share the See this URL will be good so I can bookmark!!

breedx-splk commented 9 months ago

Thanks @Abhimanyu9988 . These kinds of reports are helpful, because we also learn how people are consuming the documentation. I went ahead and opened #4030 to address the old protocol and port. Going to close this for now, but please feel free to reopen if there's more to discuss. 👍🏻

open-telemetry / opentelemetry-java-instrumentation