signalfx / splunk-otel-lambda

Splunk distribution of OpenTelemetry Lambda
Apache License 2.0
5 stars 4 forks source link

Seeing Errors when Env Variable is set to SPLUNK_LAMBDA_LOCAL_COLLECTOR_ENABLED=false #285

Open storl0rd opened 1 month ago

storl0rd commented 1 month ago

  Exception while exporting metrics HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f527b1b8590>: Failed to establish a new connection: [Errno 111] Connection refused'))

We are seeing this error on our Lambda functions instrumented using separate layers for apm and metrics arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-python:8 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-lambda-metrics:8

They are configured to send the data to a collector running in gateway mode on an EC2 instance. The local collector has been disabled by SPLUNK_LAMBDA_LOCAL_COLLECTOR_ENABLED=false , however the logs indicate that the instrumentation is still trying to send metrics to 'localhost'