search

liatrio / backstage-dora-plugin

Backstage plugin to surface an organizations DORA metrics
https://openo11y.dev/human-systems/delivery-metrics/lagging-delivery-indicators/
Apache License 2.0
34 stars 2 forks source link

Error: OpenTelemetry trace error occurred #38

Open zeshanziya opened 2 weeks ago

zeshanziya commented 2 weeks ago

We followed all the instructions and can see data in Grafana and LokiDB. However, when the Backstage plugin tries to call the API, it consistently returns an empty result. In the dora-api logs, we observe the following error:

OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (The service is currently unavailable): , detailed error message: error trying to connect: tcp connect error: Connection refused (os error 111)

Few screenshots.

  1. Grafana (Loki as source) image

  2. api error log image

  3. otelcol logs image

image

image

RobertKelly commented 2 weeks ago

Thank you for the details - We'll dig into it shortly and let you know what we find.

adrielp commented 2 weeks ago

@zeshanziya - it looks like the API itself isn't configured to connect to an OTel collector over gRPC. It's a graceful failure in that it's not breaking the API but it does mean that application telemetry from the API itself aren't being observed in your grafana stack. The DORA API should accept standard OTel Environment Variables which can be used to configure exportation to an OTel collector configured for gRPC. I'll leave this open for a bit, but let me know once you take a look based on that above information.

zeshanziya commented 2 weeks ago

@adrielp Thanks for looking into this! I’m not very familiar with OTEL setup. Could you guide me on the next steps? I followed the Docker Compose setup and made a few modifications to the .env file too, but I’m still not seeing any results.

Here’s my current .env configuration:

LOKI_URL="http://loki:3100/loki/api/v1/query_range"

# LOG_LEVEL=debug

PORT=3030

GITHUB_ORG=my-github-org
GITHUB_USER=my-github-user
GITHUB_TOKEN=my-github-pat

LOKI_DAYS_BATCH_SIZE=30

OTEL_COLLECTOR_HOST=otelcol
OTEL_COLLECTOR_PORT_GRPC=4317
OTEL_COLLECTOR_PORT_HTTP=4318
OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317

OTEL_RESOURCE_ATTRIBUTES=service.name=backstage,service.version=1.0,deployment.environment=prod
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative