search

open-telemetry / opentelemetry-lambda

Create your own Lambda Layer in each OTel language using this starter code. Add the Lambda Layer to your Lamdba Function to get tracing with OpenTelemetry.
https://opentelemetry.io
Apache License 2.0
271 stars 166 forks source link

Otel Collector does not support metric receiver although docs advertise that it does. #1425

Open bhaskarbanerjee opened 1 month ago

bhaskarbanerjee commented 1 month ago

Describe the bug OTel Lambda Collector docs say that the collector extension allows lambdas to use the OpenTelemetry Collector Exporter to send traces and metrics to any configured backend while Telemetry API Receiver document says.

The receiver is configured only for traces as seen here

Is this a miss or intentional? And why is it advertised like it is, if this is intentional.

I do see a PR in progress to add support for Logs

Steps to reproduce Added python proto layer. Added otel collector layer. There is manual instrumentation of custom metrics from the lambda code. config.yaml looks like:

receivers:
  otlp:
    protocols:
      http:

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: enterprise.otelcollector.gateway.com

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging, otlp]

processors:
  batch:
    send_batch_max_size: 100
    send_batch_size: 10
    timeout: 2s

What did you expect to see? Per the documentation, we expected it to work for both traces and metrics (custom metrics)

What did you see instead? Only traces were exported and collector threw clear error on endpoint /v1/metrics saying it has reached max number of retries.

What version of collector SDK version did you use? Version

What language layer did you use? Python

Additional context No

bhaskarbanerjee commented 1 month ago

@tylerbenson @nslaughter can you please help on my above query?

rapphil commented 1 month ago

I'm a bit confused,

threw clear error on endpoint /v1/metrics saying it has reached max number of retries.

This indicates that enterprise.otelcollector.gateway.com does not have an metrics pipeline with an OTLP receiver and that the lambda layer is trying to export the metrics.

One simple way to verify this is the case is to only have the logging exporter in the collector lambda configuration metrics pipeline and then verify in the logs that the metrics are being generated.