search

aws-observability / aws-otel-lambda

AWS Distro for OpenTelemetry - AWS Lambda
https://aws-otel.github.io/
Apache License 2.0
147 stars 56 forks source link

53612: bind: address already in use #808

Open JachMinen opened 10 months ago

JachMinen commented 10 months ago

name: Bug report about: Create a report to help us improve title: '' labels: bug assignees: ''

Describe the bug Lambda will occasionally encounter this error during initialization of environment: telemetryAPI.Listener Unexpected stop on HTTP Server listen tcp ---.---.---.---:53612: bind: address already in use When this happens, the next invocation will run successfully at first, but then time out when trying to export metrics. This only seems to happen on cold start of new environment and usually resolves itself after 1 or 2 invocations in that environment.

Steps to reproduce Unknown

What did you expect to see? Lambda runs and metrics are exported successfully

What did you see instead? Lambda times out

What version of collector/language SDK version did you use? Version: aws-otel-python-amd64-ver-1-20-0:3

What language layer did you use? Config: Python3.9

Additional context Lambda also has layers for Lambda Insights and Python libraries

drobert commented 8 months ago

We've been bumping into this, too. We're using a fastapi python app in a docker-based lambda also using aws-otel-python-amd64-ver-1-20-0:3

Xaaris commented 8 months ago

Same here on aws-otel-collector-arm64-ver-0-90-1:1

jan-xyz commented 6 months ago

@drobert and @JachMinen are you using otlptracegrpc.WithDialOption(grpc.WithBlock()) for your SDK client setup?

github-actions[bot] commented 3 months ago

This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled

drobert commented 3 months ago

Please keep this open. Unresolved/still an issue.

Joe-Woodhead-BigChange commented 1 week ago

We encountered this today. Happened twice in a row on deployment to Production where it hit a timeout on cold start. This is a .NET 8 lambda using this layer pushing data into New Relic:

arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-collector-amd64-ver-0-102-1:1

Error below:

{ "level": "error", "ts": 1729787819.487711, "logger": "telemetryAPI.Listener", "msg": "Unexpected stop on HTTP Server", "error": "listen tcp 169.254.79.130:53612: bind: address already in use" }