search

newrelic / newrelic-lambda-extension

An AWS Lambda Extension to collect, enhance and transport telemetry to New Relic for AWS Lambda functions without requiring an external transport such as CloudWatch Logs or Kinesis.
Apache License 2.0
74 stars 50 forks source link

Takes super long to execute #90

Closed navihtot closed 2 years ago

navihtot commented 2 years ago

On some occasions it prolong my function execution to maximum time

Steps to Reproduce

not sure. a lot of invocations in a row.

Expected Behavior

Not to prolong my function execution for so much and cause unexpected costs on AWS. If it cannot deliver a message it should give up much faster.

Relevant Logs / Console output

    2021-10-26T11:17:49.299+02:00   [NR_EXT] Request failed. Ran out of retries after 3 attempts.
    2021-10-26T11:17:53.697+02:00   [NR_EXT] Telemetry client error: Post "https://log-api.newrelic.com/log/v1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    2021-10-26T11:17:56.579+02:00   [NR_EXT] Sent 0/1 New Relic function log batches successfully in 241457.663ms (35517ms to transmit 0.0kB).
    2021-10-26T11:17:57.238+02:00   {"v":0,"level":20,"name":"newrelic","hostname":"169.254.63.37","pid":17,"time":"2021-10-26T09:17:57.098Z","msg":"Not recording function setTimeout, not in a transaction.","component":"Shim","module":"timers"}
    2021-10-26T11:18:06.003+02:00   [NR_EXT] Request failed. Ran out of retries after 3 attempts.
    2021-10-26T11:18:10.465+02:00   [NR_EXT] Telemetry client error: Post "https://cloud-collector.newrelic.com/aws/lambda/v1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    2021-10-26T11:18:12.298+02:00   {"v":0,"level":20,"name":"newrelic","hostname":"169.254.63.37","pid":17,"time":"2021-10-26T09:18:12.159Z","msg":"Not recording function setTimeout, not in a transaction.","component":"Shim","module":"timers"}
    2021-10-26T11:18:13.761+02:00   [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 52017.034ms (38797ms to transmit 0.0kB).
    2021-10-26T11:18:13.898+02:00   [NR_EXT] New Relic Extension shutting down after 26 events
    2021-10-26T11:18:14.061+02:00   [NR_EXT] Sent 1/1 New Relic function log batches successfully in 16302.653ms (179ms to transmit 0.4kB).
    2021-10-26T11:18:14.099+02:00   [NR_EXT] Log server terminating: http: Server closed
    2021-10-26T11:18:14.100+02:00   [NR_EXT] http: panic serving 169.254.79.129:46010: send on closed channel
    2021-10-26T11:18:14.100+02:00   goroutine 30 [running]:
    2021-10-26T11:18:14.100+02:00   net/http.(*conn).serve.func1()
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:1801 +0xb9
    2021-10-26T11:18:14.100+02:00   panic({0x7e4300, 0x942dd0})
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/runtime/panic.go:1047 +0x266
    2021-10-26T11:18:14.100+02:00   github.com/newrelic/newrelic-lambda-extension/lambda/logserver.(*LogServer).handler(0xc00036d5c0, {0x94ec90, 0xc0000f4620}, 0xc0000a6700)
    2021-10-26T11:18:14.100+02:00   /home/circleci/project/lambda/logserver/logserver.go:164 +0x22a
    2021-10-26T11:18:14.100+02:00   net/http.HandlerFunc.ServeHTTP(0x4b4fe6, {0x94ec90, 0xc0000f4620}, 0xc0000f4620)
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:2046 +0x2f
    2021-10-26T11:18:14.100+02:00   net/http.(*ServeMux).ServeHTTP(0x0, {0x94ec90, 0xc0000f4620}, 0xc0000a6700)
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:2424 +0x149
    2021-10-26T11:18:14.100+02:00   net/http.serverHandler.ServeHTTP({0x94de78}, {0x94ec90, 0xc0000f4620}, 0xc0000a6700)
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:2878 +0x43b
    2021-10-26T11:18:14.100+02:00   net/http.(*conn).serve(0xc000108aa0, {0x950580, 0xc00026c030})
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:1929 +0xb08
    2021-10-26T11:18:14.100+02:00   created by net/http.(*Server).Serve
    2021-10-26T11:18:14.100+02:00   /usr/local/go/src/net/http/server.go:3033 +0x4e8
    2021-10-26T11:18:14.204+02:00   [NR_EXT] Sent 1/1 New Relic function log batches successfully in 143.268ms (142ms to transmit 1.0kB).
    2021-10-26T11:18:14.204+02:00   [NR_EXT] Extension shutdown after 813163ms
    2021-10-26T11:18:14.205+02:00   END RequestId: d43867d8-479d-4978-88d1-7510bed882d0
    2021-10-26T11:18:14.205+02:00   REPORT RequestId: d43867d8-479d-4978-88d1-7510bed882d0 Duration: 300699.81 ms Billed Duration: 300000 ms Memory Size: 512 MB Max Memory Used: 512

Your Environment

kolanos commented 2 years ago

This should be addressed by #76 and #101. Let us know if you continue to see this issue after upgrading.

thiagosardinha commented 1 year ago

@kolanos this is still happening to my lambdas using the serverless-newrelic-lambda-layers plugin

the ARN it's using is the latest java 11 in eu-west-1: arn:aws:lambda:eu-west-1:451483290750:layer:NewRelicJava11:23

I see that in us-east-1 the latest version is 31 instead of 23. could this mean that, because of the region I'm in, I don't have this fix available? if so, how can I know when it will be available in my region?

Relevant logs

2023-01-26T10:00:48.165+00:00 START RequestId: 18103e9c-e54c-41b0-ae70-8db5c1267ca0 Version: $LATEST
2023-01-26T10:01:18.526+00:00 [NR_EXT] New Relic Extension shutting down after 16 events
2023-01-26T10:01:18.727+00:00 [NR_EXT] Log server terminating: http: Server closed
2023-01-26T10:01:18.853+00:00 [NR_EXT] Sent 1/1 New Relic payload batches with 1 log events successfully in 125.897ms (125ms to transmit 0.4kB).
2023-01-26T10:01:18.853+00:00 [NR_EXT] Extension shutdown after 2441981ms
2023-01-26T10:01:18.854Z 18103e9c-e54c-41b0-ae70-8db5c1267ca0 Task timed out after 30.69 seconds
2023-01-26T10:01:18.854+00:00 END RequestId: 18103e9c-e54c-41b0-ae70-8db5c1267ca0
2023-01-26T10:01:18.854+00:00 REPORT RequestId: 18103e9c-e54c-41b0-ae70-8db5c1267ca0    Duration: 30689.46 ms   Billed Duration: 30000 ms   Memory Size: 1024 MB    Max Memory Used: 220 MB
mrickard commented 1 year ago

@thiagosardinha AWS creates layer version numbers by incrementing from 1 when the region first received those layers. Exact numerical version can't be maintained across regions, so the "latest version" in each region might not have the same integer suffix.

There's a team working on a next-generation version of the extension, though it's a work in progress: https://docs.newrelic.com/docs/infrastructure/amazon-integrations/aws-integrations-list/aws-lambda-telemetry/