search

newrelic / newrelic-lambda-cli

A CLI to install the New Relic AWS Lambda integration and layers.
https://newrelic.com/products/serverless-aws-lambda
Apache License 2.0
47 stars 53 forks source link

403 Forbidden error in the CloudWatch logs #229

Open pjhavariotis opened 1 year ago

pjhavariotis commented 1 year ago

Description

I have already installed the integration by using a USER api key created for this purpose in my account. The only error I got from this step is the following: "Could not enable New Relic AWS Lambda integration for Cloud integrations account [New Relic AWS Integration - xxxxxx] in New Relic account [yyyyyy]. Please contact New Relic Support for assistance."

Nevertheless, I proceeded with the installation of the layer to a specific function in my AWS account. Unfortunately, I could not find any related data in the Logs or Metrics & events. I'm wondering if this issue is related to my account (should I upgrade it ???)

Steps to Reproduce

Expected Behaviour

Relevant Logs / Console output

In the CloudWatch logs of the lambda, I realized the following error: [NR_EXT] Telemetry client response: [403 Forbidden] {} [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 272.606ms (272ms to transmit 0.4kB).

Your Environment

Additional context

pjhavariotis commented 1 year ago

In order to eliminate the possibility that the function does not have permission to read the secret as part of its execution role, I explicitly defined the NEW_RELIC_LICENSE_KEY environment variable with the appropriate ingest key value. This resulted in the resolution of the 403 error. Now in the CloudWatch logs I can see the following error: [NR_EXT] Startup check failed: There is both a AWS Secrets Manager secret and a NEW_RELIC_LICENSE_KEY environment variable set. Recommend removing the NEW_RELIC_LICENSE_KEY environment variable and using the AWS Secrets Manager secret.

pjhavariotis commented 1 year ago

Finally, it seems that the 403 error is here to stay! I strongly believe that this is related to my current account plan. Can you please verify that?

Here is the latest log output: 2023-01-26T08:54:37.519+02:00 [NR_EXT] New Relic Lambda Extension starting up 2023-01-26T08:54:37.525+02:00 [NR_EXT] Using license key from environment variable 2023-01-26T08:54:37.527+02:00 [NR_EXT] Starting log server. 2023-01-26T08:54:37.651+02:00 [NR_EXT] Startup check failed: There is both a AWS Secrets Manager secret and a NEW_RELIC_LICENSE_KEY environment variable set. Recommend removing the NEW_RELIC_LICENSE_KEY environment variable and using the AWS Secrets Manager secret. ``2023-01-26T08:54:38.227+02:00 [NR_EXT] Sent 1/1 New Relic function log batches successfully in 575.994ms (573ms to transmit 0.3kB). 2023-01-26T08:59:41.428+02:00 [NR_EXT] Telemetry client response: [403 Forbidden] {} 2023-01-26T08:59:41.428+02:00 [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 259.005ms (258ms to transmit 0.4kB). 2023-01-26T09:05:17.300+02:00 [NR_EXT] Telemetry client response: [403 Forbidden] {} 2023-01-26T09:05:17.300+02:00 [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 874.769ms (874ms to transmit 0.4kB). ``2023-01-26T09:11:06.788+02:00 [NR_EXT] Telemetry client response: [403 Forbidden] {} 2023-01-26T09:11:06.788+02:00 [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 274.653ms (274ms to transmit 0.4kB). ``2023-01-26T09:17:32.067+02:00 [NR_EXT] Telemetry client response: [403 Forbidden] {} 2023-01-26T09:17:32.067+02:00 [NR_EXT] Sent 0/1 New Relic payload batches with 1 log events successfully in 290.011ms (289ms to transmit 0.4kB).

chaudharysaket commented 5 months ago

Hi @pjhavariotis

  1. [NR_EXT] Startup check failed is a warning message that indicates that ingest key is stored in multiple places and recommends only one place. It does not indicate any error.
  2. [403 Forbidden] {} is response from newrelic backend that it received a request without proper api-key or newrelic-ingest-key. Such issues are mainly due to lambda not having proper permission to read secrets.