Getting UNAVAILABLE: Socket closed while connecting to Elastic APM Server

[infa-ahossain]

We are trying to perform a POC to get nginx traces to Elastic APM Server using the grpc protocol.

otel agent -- > APM server is not working and giving the error UNAVAILABLE: Socket closed

although the flow otel agent --> otel collector --> APM server is working

Both the agent and otel collector are deployed on same K8's env.

Otel collector configuration looks like based on example https://github.com/esigo/nginx-example

collector.yaml

    exporters:
      otlp:
        endpoint: "<apm server url>:8200"
        headers:
          # Elastic APM Server API key
          Authorization: "Bearer <token>"
        tls:
          insecure: false
          insecure_skip_verify: true

Steps to reproduce Try to install ingress-nginx and point the endpoint to APM server

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx --create-namespace --set controller.opentelemetry.enabled=true \ --set controller.service.type=NodePort -f ingress/controller-config.yml

controller-config.yaml

controller:
  opentelemetry:
    enabled: "true"
  config:
    log-level: "debug"
    enable-opentelemetry: "true"
    opentelemetry-operation-name: "HTTP $request_method $service_name $uri"
    opentelemetry-trust-incoming-span: "true"
    otlp-collector-host: "<apm server url>"
    otlp-collector-port: "8200"
    otel-exporter-otlp-headers: "Authorization=Bearer <token>"
    otel-max-queuesize: "2048"
    otel-schedule-delay-millis: "5000"
    otel-max-export-batch-size: "512"
    otel-service-name: "nginx-proxy" # Opentelemetry resource name
    otel-sampler: "AlwaysOn" # Also: AlwaysOff, TraceIdRatioBased
    otel-sampler-ratio: "1.0"
    otel-sampler-parent-based: "true"

What is the expected behavior? The Agent should be able to send the traces to APM Server as APM Server also support the grpc protocol

What is the actual behavior? We are getting this error

[Error] File: /tmp/build/opentelemetry-cpp-v1.8.1/exporters/otlp/src/otlp_grpc_exporter.cc:67[OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: UNAVAILABLE: Socket closed"

Additional context Add any other context about the problem here.

[NicklasWallgren]

Did you manage to solve the issue?

I'm getting the same error while trying to connect using the nginx module.

/opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:135[OTLP TRACE GRPC Exporter] Export() failed: failed to connect to all addresses

[infa-ahossain]

no but as the solution of using otel collector in between works so i am using that

otel agent --> otel collector --> APM server

[NicklasWallgren]

no but as the solution of using otel collector in between works so i am using that

otel agent --> otel collector --> APM server

What otel collector are you using, if I may ask?

[infa-ahossain]

try using this example - https://github.com/esigo/nginx-example/tree/main it has the one collector and demo working