search

newrelic / newrelic-lambda-layers

Source code and utilities to build and publish New Relic's public AWS Lambda layers.
https://newrelic.com/products/serverless-aws-lambda
Apache License 2.0
34 stars 42 forks source link

API Gateway 502 due to Lambda Proxy body is null #141

Closed lucasklaassen closed 1 year ago

lucasklaassen commented 1 year ago

I've installed the serverless-newrelic-lambda-layers plugin in my nodejs18.x serverless project ensured i've set the NEW_RELIC_LAMBDA_HANDLER variable for each function and verified that the NR settings are working properly.

I'm running into a problem with API Gateway not receiving the body from my callback when using this integration.

Endpoint response body before transformations: null

It appears that statusCode is being passed through properly but for some reason the body parameter is being set to null.

I'm using async lambda handlers and returning like this:

  return {

    statusCode: 200,

    body: JSON.stringify({

        responseObj

    })

  };

When I remove the newrelic layer plugin and deploy the function it works as expected through the API Gateway lambda proxy set up. It seems like the way the newrelic layer/instrumentation within the newrelic-lambda-wrapper.handler isn't handling the body parameter properly. Is this a known issue? I've spent the past 2 days reading through all of your documentation, going through all of your github repos and searching issues and PRs and I cannot solve this for the life of me.

workato-integration[bot] commented 1 year ago

https://issues.newrelic.com/browse/NEWRELIC-7193

lucasklaassen commented 1 year ago 
timestamp,message
1678053138619,"INIT_START Runtime Version: nodejs:18.v5 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:c869d752e4ae21a3945cfcb3c1ff2beb1f160d7bcec3b0a8ef7caceae73c055f
"
1678053138692,"[NR_EXT] New Relic Lambda Extension starting up
"
1678053138697,"[NR_EXT] Registration response: {""functionName"":""aws-nodejs-production-poolEventFightsList"",""functionVersion"":""$LATEST"",""handler"":""newrelic-lambda-wrapper.handler""}
"
1678053138697,"[NR_EXT] Using license key from environment variable
"
1678053138704,"[NR_EXT] Starting log server.
"
1678053138704,"[NR_EXT] Log registration with request  {""buffering"":{""maxBytes"":1048576,""maxItems"":10000,""timeoutMs"":500},""destination"":{""URI"":""http://sandbox:44633"",""protocol"":""HTTP""},""types"":[""platform""]}
"
1678053138704,"LOGS Name: newrelic-lambda-extension State: Subscribed   Types: [Platform]
"
1678053138705,"[NR_EXT] Registered for logs. Got response code  200 ""OK""
"
1678053138966,"[NR_EXT] Startup check failed: There is both a AWS Secrets Manager secret and a NEW_RELIC_LICENSE_KEY environment variable set. Recommend removing the NEW_RELIC_LICENSE_KEY environment variable and using the AWS Secrets Manager secret.
"
1678053138966,"[NR_EXT] Sending function logs for request 0
"
1678053139142,"[NR_EXT] Sent 1/1 New Relic function log batches successfully in 175.889ms (174ms to transmit 0.3kB).
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.005Z"",""msg"":""The native-metrics module is disabled by default when serverless_mode is enabled.  If desired, enable the native-metrics module via config file or environment variable.""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.005Z"",""msg"":""NEW_RELIC_NO_CONFIG_FILE set, deferring to environment variables.""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.006Z"",""msg"":""Using New Relic for Node.js. Agent version: 9.10.2; Node version: v18.13.0.""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.142Z"",""msg"":""Using LegacyContextManager""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.145Z"",""msg"":""Adding destroy hook to clean up unresolved promises."",""component"":""async_hooks""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.184Z"",""msg"":""Agent state changed from stopped to starting.""}
"
1678053139352,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.185Z"",""msg"":""New Relic for Node.js starting in serverless mode -- skipping connection process.""}
"
1678053139354,"EXTENSION    Name: newrelic-lambda-extension State: Ready    Events: [INVOKE,SHUTDOWN]
"
1678053139355,"START RequestId: 275f6c77-c942-461c-a63d-248737f89225 Version: $LATEST
"
1678053139356,"[NR_EXT] Aggressive harvest yielded 0 invocations
"
1678053139488,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.487Z"",""msg"":""Harvest started.""}
"
1678053139500,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQSsxYTZXL2JSaGIvVndSaFA3UmRpWnlUTTFUUUQ5NGNUZENrRFZiZTdRSkJJUXlIUTRrMVJUSThmTUwvKzc0aFJWdXlKRnQycE1SQUtoOGNEUi9mK1pzM2I1NTAxWitiU29XcVV2M1JWVjhWYVg5ay80L1VXVGxLMUR3STFhZ3VoMmVtcklaazVGSHBDY1FsbGxTT29qclZWWncxcE1NMEM4MWY1VEF2c3JCdVpvZDVsaVd2VDAxYXZZbW5zNnA4SDVkVmY5RHZIcHFjbXFLRUswajd4L3VqNDlmalk3aHB6bzJ1bTdzbVBZMkxMSjNENDBCeDlNZDQ4cjdSWmRMS3dkSTVCM3FRVm1VNlMyNlpZVy9RVjFONGFvbS83MkRra1A3MW9OOVpDUllYc1o2MGJ6K2xkWklNc0NjazRoUlRvR2JMNzVpVWcwK2ZydnFwbWh0Z05xN3pQQ3NxRmNSSlhGMjR2elhxdVA5dGhibFlncGhQZUlDNm56Ly9IRHowN0ZHZUo3RlcxbXozOTl5a2NUcDFYOVZGTTdIZzVoRGtDY0tFWUR1T0VTT0Nja3dKbGh5dVNENVNqM2RwWE1VcWlTL2J0MnZxTUxyeFArSStZdjU5c3Q0WVZkV0ZLVUZDV1JXMWplL0M4dlRmNW5NZEY4Yk5wNC93NFU3OGJzTHptT2djZlh6bmxxWnFZZmRXcFdGaWlzY0g5N2hRYWFrYXhKZnUyRXl0Z21YTEJvdmJYeUx2OWRvNFYybXpsTnpqckZMdHFpckh4clFCUWJ0cHRJVUpMTEQ3bVB4aGdpVWJPZ0JnZ2lUM2tMUmpKQm5sUWtqZmg0SGdkKzZ1VUdMbUlRLzdoTUpBd0RTK1IxU2o1SEVNZDI1bCtwd3h3WkJIK0RhNUt4UnJUNEJZWXNtRnoyRjV3R0JGL3R1cXlsL0ZaYTRxUGJ1SjlkY3cxWDJ6U0lydTdwbjBHd1RpeTlROGJPeTZEUFd2aTVjcWdaWHEvaWM5U2JPemRPc1ZxQTdxdzZjb0JINC9xRTVIZVdqT254aEZOR2dTbjRQYnYzVytEeEc5Z3MyMnJESkl5Rmx1V3QrNEg3T3ltdHJzWFpyRTZFWk9zM3NSektXVVB2TVhMN0YxRW1HQ3FTQmVZenpzYzlTanBKbEhXSHJnTCtRVDJNdUpMN2JvY3V2MUJtU0VVQ3FrOE82T0VSSk0rZ3NmM3hFak9PUElZMUFXZVZ4dUVYTmo2RGVRZDNCaFgwTkdDZHU2c1p2bnJXbDVIVUM5NHRRbGJPNUxBR3BFSUNFOGlhU0h1aGZiMHl4Q0hxS01ZZzloUktrUGx5MEt4MUNQcUZRdmhXSStWOE5JUVgxU1hqaTZLaTVWa09yTEhDWE9UWG50RkdIcHFMbTZ6RkpZbTQ3TzVpNW5sQnpjdVJzV1pGWXRiWUVyQWg0Y2M4ZER2aUNjZ2g3WTZtQ0cyMWJmcHFoT0ozYTJxYzBEVlpxMXlHSk9LT083aitGRktHd21pRURxWU1ocVJCOE5zcWpKaFhkMVdmYjFMbU9iclloa0hyaUpZc3dJZTdRZU5qbFBMT0lualVhVFBOWW42K0JmeW9rN2phMWlUTUJwQjBsRWtiQ1lKL2VXeHFsS0xxcFlsMjBOdWJFbzNiSEl2bzlUV3QzSDZkRzVZZEF2TlNCOW45WFhNOGd6VzlmdVFjejk5bm5nSUdZOWc1eHlDTHVlV1g0NlRPajJrdXZBMEQ0Yy9OS0pzWEpXR2tOWGZiRFBGS2RaWEV6SytCTFVKL0RNb045UWxwUFNKcndSUnRkTms2aTZhT3l6UjIwd3R5cVVOdTlDbU5DRWhvRUJzRkNrRmZNalJLUmdvVlRHSU02SjlvQjRXc2VXRWtlKzRVb0hWSGtjTVNSc2wwc1ZJS3JoNHd0a0pFR0I5SVdpa3VOV1NPZklob1FTcGVCQkduRWVHUndvNjNZMXp4TURONnVpTnJackZtY0ZaRjlRMnlGWU1zWUhYWXp0V1NWUSttVFVNL080ZXA5bEozVU9ERFRFZTVvVjhFUi9hbEpUeE5vS2hpTWhJR0dlMjY1YjF5NkRtbVRRRDd0MjBhanhPTFBMZzEwUHJ1QzNueFlUYzY2VHVveFB6YVFqbk16akpJbExTNzhnYm9EMzVkNDBPbUtha1JDRlN2SW9DdmJ0cnk4SDhxT2M2K0ZWNTk2ZS9jQ3ZEakF1TGo1bXNXMlp0cG92MmVyc3NmUFFCTEl3bjJ0UTBabWJhcFpaUi83eTJyWnd1K202aUdIT3RReEtsL2h1dTF4Y1dJb1cwcmE4Ylp2THpzRTd6bGJXUXFzbTRFVHd5Tk5DRExYUHlKQjVXQStWUjhNaFlWSlFFVW1mRUw2cVlxUDdPS3NMYmRyeGNRdE1sY2UvUVBETzFNVjJlcVYxVmk5Vzc3SXQ5enlSeHcyMWpQNVNaVFlOcU5sT0M2bXBHYlRMSXFFRkRSOG0vcWlxMlcxc3J1THd1b3ZPVlhOOUYxNXZad0tnbkZyYmI5MitTcXV6SkJ4WHFyakI0S3lxY3Z0VVZaY3ZJV0kyQUFndEFXVm1WQWhWbW5XVXlXMnp2ekxubFR1cjVzbEFMWFdreiszTVA4L3Z6czZURjU5L1JvNC9pT2VnbDZ0TzQyZ3hQRE5CM3MzbTZYVHdrL3RUUXlwWEdKVHhORFhoRUZMU1RLVlQ4K0wwNTRBMlpHS0RpblpqTzVxMkgwbDh5QzRoWnltWE82ajN3d2VsWWRWbDVleEY3MTFhbWFRSEU3M2Z4NzMvOVRDYVlENFJQL1pzZDkzQTR2czFybHhPaFVPOTNnKy92ajMrOEg3UVMrSVQwL3ZGNkpQc3g5N0xXWkhOalF1N3N0UDg5TVlxVWtXOGVHU0RUalBZZ1ZmQTRyUWZvcGdoNEdqcGFMbHlyT3dQSHNqRGtqbkxMYWQ5cGVOUUlVNFVEa0lNbTFjWWt0WE43ZERKMnVaVThGaHBLdHZOek9ySEpWOG04ZHJPaG56cCtUdHViQzN0dnZZMUh5TkJ1YWRsb0wxSW9hL3J5RDFVZ2t0K0R6dHVkanFiNTFuYXJyQ080L2FBb0xzQjZRcTlwcFp6VHVLMFRZdHhlNVRjSVV6VzFKWkRHRGhkQ3dkNHpPZHEwYnF4bmphbWFOYmR3aDFQYWV4MGJGUVlnb25sRTdtTTJ2YlFua0FsUTJLbzRqN25XSWFhNHVjSHFydEhwbjJneUJOM1VIUnpJbm82aXJEVHNmZ09ZUlFKaXBEV1V0S0lCcjRTendsR1c1c0tlMEVTWGN0SGk1N0JGK1NqanNQM0NDU2ZDNE94VDBFcEJmKytGWkR1YWJIdEJUYnJkY1dpZy9aazJFRDV1dUR3SGNJR29SQ3dvYlZtUG80NDhaOVQvdG5ZbHQ0SGlQaGE3bG52S3o4WlRzSnBlWDJYY01MY3c3NW5CQWtJanI3NmRwYWF5aGxuK2dRdU9rdlROY0E4MkRGaWQzRkJCTVpRNGUyNC9maHdBcFJ3MXBDWTdPMFV1S0dKdWVUUkRmN2VxMGZEdEhTU0p6UTJQVyt0VGhRZUlvanVXaEl1cUp0ZXMxcDg3clZUdjlsK3ZMRFdjRzc3eldjbWVMWFNEL1RndUM0dzNXWUVGM3R0WFlaYlpOOThkV3QwOTRzOEhYeVd2MU1HcGhWRlZ2UkhrVXBLMDM1TDAxWm55NWJkZnQ1OWM5djJxbC9hM2xwL3hKOEd4QTA0MmdTY0ZXenRvL3Y1ZCt2cm03YSsvbTQrZitmTlo1dURyNi8vRCtJVnZKTDVMZ0FBIl0K
"
1678053139500,"[NR_EXT] Aggressive harvest yielded 1 invocations
"
1678053139500,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:19.500Z"",""msg"":""Harvest finished.""}
"
1678053139709,"[NR_EXT] Sent 1/1 New Relic payload batches with 1 log events successfully in 209.422ms (208ms to transmit 2.5kB).
"
1678053139710,"END RequestId: 275f6c77-c942-461c-a63d-248737f89225
"
1678053139710,"REPORT RequestId: 275f6c77-c942-461c-a63d-248737f89225   Duration: 354.90 ms Billed Duration: 355 ms Memory Size: 1024 MB    Max Memory Used: 118 MB Init Duration: 735.58 ms    
XRAY TraceId: 1-64050f12-6947e176797519f723aac436   SegmentId: 1ecb06457e134546 Sampled: true   
"
1678053142919,"START RequestId: ce2a56ae-ad73-471d-a0a0-53367e08b633 Version: $LATEST
"
1678053142919,"[NR_EXT] Skipping platform log for request 275f6c77-c942-461c-a63d-248737f89225
"
1678053142919,"[NR_EXT] Ripe harvest yielded 0 invocations
"
1678053142927,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:22.927Z"",""msg"":""Harvest started.""}
"
1678053142929,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:22.929Z"",""msg"":""Harvest finished.""}
"
1678053142929,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWjIyN2JPQkQ5RjJFZmJaa1g4ZWEzYmkvQUFzVnVnV1N4RDBFaFVCVHRhaXRMWGxKS21nVDU5dzRsSzNYaXhIRVNPMm14c21IVEVqVnpabmlHTXlOZlJndmI2RnczT3BwZVJ0cFYwVFI4VHZXWm41WjZrZVY2MnZyeG1mWE5tRXc1bFZ3Z0pyR2tjanBySzlNVWRUZDFYTlc1L2RlUGw2N08yKzdzZUZuWDVmdFRXelVmaXZtWHhuOHNmQk9Ob3VHbTlOUTZEOStnN2JlUGI0N2ZIeDNEUmZ2Tm1yYTdhcXZUd3RYVkFtNkhHVy8rT1VvL2RsalNYZytXOFRlWUQ5cWEydFRsRDJHWWp5STloN3ZXNUtzWW81aEVWNk5vc0JJc2RvVkorNThuVlZ1V0k4eUZSSXhpcXVKRXl1dWZDWWtWRWFPVGs4dW8wZ3NMMG83YTViSjJqYzZLc21qT0ozOTJlQ2JIVGxkZWQ1YjV5WkdkQitBZVZKN2dFVjUvRS96NTgraGVZVWRMWFhVdW14elhqZTY5NTQrc3JZSWtOa0xENndreXdJOWJaUHpWZkxGdXpZYkpoOVV5VFhaZjI4NVlGQ1BFRTZFNEk5MllBRjBrVGpCU1ZDUktLYjR4WTMyTVVJS1ZaRktGQVVaVUtiWVZabWZoY2JHd3o4TXJFeTRTZm8xeEcvQzdwdDU1RWl6Z1VnaEZDWkdjSkppclpMdkhkVm0rcWd0ZjJSL3ZXcWNEbnQvUDM0SW5ySnY4WFgydDZyUHEzdThYOHRkVGNIVStmaEZ3c0lINXBuWjJVaTl0ajNQeXFmYk4zRmsvOGJhMHBxTTU3WVNCenhrbWZHT01KR1VLaTI2TVpZSTRZeXZsaUZORUdDd2tZS2IzS0Y0Mzk1QjZyczE2RFlVdjRNUkRxL0NOYm16SVNqOE1XN1paV1ppNDlaQXAxOWlDTnpUc01vYURVaWtSUnBJb1FCTzRldytVb2dJd2xWbHo4V0toeHpOZHdZVHoyRFR1UW1lVnVWaWlNcjZ1UEdLWCsxZ3Y5RVZkd1E0Zm0zb3hZUWtscitlMVdaZElOdDBHeFFOanljNWpPQWpCRFBaRjhKekMvTkU0UW1aTHd4S21IYUowV1ppdm03RFd2ZkRRbU1kS0NpWUlSVEpzUEFqYnNkaFdjTHlwZEhuZUZNYjNwY1pkcFF2ZXJYVFpLcWxxdGtsNk5ORkhrVGV3YWU2NzlqbGcyT3pLeFVOWjlsTXgrMEJHUGo5T3dNeklRd21lMnFEalJvdHhHWUYxMXAzV2hVdDljUUhnQ2R3R1hVL0g4TlNIWUpteXE2N1phTTQ3NDBJeEQ3WTJUaHY3Unc0bkdEWlcwTndTemhKcGpBUXNNOHZ5SkxmV1dDVW9USjYzUlppWk1JSXpyVVNTVVUxNGtvVjJTVHZRMU1teG1jbVZTWmptZ2ljY29WN0o0TU51aXRTSVpFeklqQVM2V2hGOHJoZkwwc0xGeHJVMnRGOUY3U0J3b2UyS0ZaUjNUSXllRVk4RzVzMXJCOUs2UHEyN081eXVGOHU2NnZ2QVFVSkFDeVVyaUY0c1E4ODM5R3FLWUdqeVZoVmJOUDBSYktIVVdoMzk4c1JmaXlvWWFjcWkzMWt1NFIxVkxyWGZUTm42NHRTbWc1eDBVWlJsNFlPUnE4Q044aXdla2hqSWdPeTFTbDdCeDlhNitBdmdYRG5pS2FsdEVLUHpISXoxVDVReTdSTmtGM2JQcDlPaENiT1BNTDdCb2JtdExEVDU5MU1GM2FiS1VIa0REV0lRN000LzFVV2dYWTk5emRwNHIxMXp4N3l3WnYzVGx2amdqMkNDTG1mL2EwRmV0MjdHRXMyNHRtT2RDenBPQk03SEdtazBacFJ5WVpITU9BMUUyQjRjSkw3VjJZaDlNVTh4QUlFektTU2pYREh6c2h0WklJL3pzYmROYUpMcjlyRXNFN2RaaGtLdVIzeW5IUWZtOTVQMzVFcEtWVExUV2t0c0JXWk0vbnc1NFlHa3Y1Y2N3VGVXWkpYUm41d1pVRHhJK0I5bUJzSzF3WmJZM013eUxKTFp6MGVxMi9YeFhsaVViRlFhcStMM0dmWEZJT0dYWWxHb2VQV3FjOXVwNnNWb3MrekZmZGw3WnJOM04zT3lGQnhLTHJRNjhMWnN2dGRLSW44SXh2VnowK21kenp5SHdGb0RFdjdnY0s1MjBYU21TMi83dnlNeTdlMU5rNGVuRjllWHc0UEh0M1ViZUVtZkZxSjNSTmhkSVhVajZuNlZ3aVFROE9ycU80Q25kOS9HR2dBQSJdCg==
"
1678053142929,"END RequestId: ce2a56ae-ad73-471d-a0a0-53367e08b633
"
1678053142929,"REPORT RequestId: ce2a56ae-ad73-471d-a0a0-53367e08b633   Duration: 10.13 ms  Billed Duration: 11 ms  Memory Size: 1024 MB    Max Memory Used: 118 MB 
XRAY TraceId: 1-64050f16-21b0b3edb4c5a08790a838ab   SegmentId: 0a7d594e7636be2b Sampled: true   
"
1678053143830,"START RequestId: ac3da8b0-899e-4647-852e-e9961e353424 Version: $LATEST
"
1678053143841,"END RequestId: ac3da8b0-899e-4647-852e-e9961e353424
"
1678053143841,"REPORT RequestId: ac3da8b0-899e-4647-852e-e9961e353424   Duration: 11.13 ms  Billed Duration: 12 ms  Memory Size: 1024 MB    Max Memory Used: 118 MB 
XRAY TraceId: 1-64050f17-b1e7f47b87554ea263ac37f1   SegmentId: 20862a2334c2354a Sampled: true   
"
1678053143843,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:23.840Z"",""msg"":""Harvest finished.""}
"
1678053143843,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:23.839Z"",""msg"":""Harvest started.""}
"
1678053143843,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWmJVL2pPQkQrTDlGOWJGTTdUaHk3MzdoOWtVNUNkeXZCNlQ2Z1ZlUWtic21SeEQwN2dRWEVmOSt4MDBCcGFTbWxCVllYV3VvbXpzd3o0MmM4TSttdFY4bEc1S0lSM3ZqV0U3cjJ4dmIvV0Z5WmNTbXFOQmZqMWd5dnBHbUd3WmdTUm1NVU1jd0lHMC9hT21zSzVhWU9hNVhMZjgxd3BsWGV1clBEbVZMbGwwdFpOMStMNlhsampndlRlQU92dnltNWxOckFKMmo3N2ZqbzlNdkpLVnlVUDJUV3VxdXl2aXkwcWl1NEhXWWMvWE9TSERzc1NhY0hNLzhIekFkdGpjcFUrU0FNMDRFbnBuRFhnbnp1WStRSDN0M0E2NjBFaTNXUkpkM1hzN290eXdHbU1VTVJ3V0hnODRBOWZDVStJM3h3ZG5icjFhS1NJTzJrbmMyVWJrUmFsRVZ6UGZyVDRSbWRhbEViNFN3em94TTV0Y0FOcUR6REE3ejRDdkQzNzRPMXdrNW1vbll1RzUycVJuVGVNeWRTMWxaU05FRDkzdzR5d0k4YlpQelZuRXU5WU1QbzYzeVpSdHV2clRNVytRakZJV2NrWm01TUVNRUlIQnFTT0ZxNXVEaEdLS0pCRU5FNG9BUXhGbTZFNXl3N0xTcjVlcHlSd3haeU9EWUFYcDcxNU1tNUNRR1BDT1VrSnZGbUY0dXlmQmVmdlpjRFByZGFXQ0MvWDM4QzA2VWUvVjFmMU9xcVh2dDVZQWZ0Z3NjNTlhQ2dZRXN5amRKeXBHYXl3emY2cGt3ejFkS01qQ3hsNWdoTW5LUXdqQmpzeFN0alRIaU1jZHlOS2Fac1BvYUQwZ2lBTWhUSE1Zdmg2aHJsaTZZZVd0ZTllZStsOUkwYytoWnFUQ01hYVhQUGc0R3pOaTJMekc4TjVNTUZCajJoY1lzeEhKaEhtSk9JWThESDE4QW9hZ0JTWnd0dXJpb3huSWdhSmx6N1dhTnZSRnBuTnpOVSt2ZTFoYTl6NDR0SzNLZ2E5bkkvVTlVb0Nrbnd2bDZidUxTeDZyWXdER2tZYnoyR0kwQVdRY2lqa0NKdXY3OFlpODFsaVYzR3hLRktaa1YyOFFTMEJROXNNN1pRS0E0Ump3T013NlVWWFNvdGptcFJYamRGWnJxaTRxa2lCVzlYcEd5VVZEZWJKTDJZN0FQUFpMQ1o3cnZLT1ZEb2JNdkZRMW4xNFpoOUtFTmZIU2RncG1lZzJFNmsxZkdvbWJqMXdEcXBMMVdoRTFQY0FQZ0E3b2YreGpFOE1UWll4dEdkYXl1YWEyZWNMZHZCMWthTFRQNlJ3d2tTUzVhSkRLVW9KbG1BR1U2REVKR2NUbkNZNDRCUW1EeHRDenR6UWttV2hvTGhpY1E0eW9sdGpJUUdUVTZPRUN6aU9RdEZ6aWpCbUhWS2VoKzZLUlBDUkJEbHVVemhQWW5zRkNPcVdTbmhZcU5iYVJ1dFFta0lYR2l3ZkVZRGlxRE4yajBlTTVnM1ZScWt1WTdNM1cxUHEycW02cTdqNnlWWXRGQ3pndWhxWnJ1N3ZpdGpKSUIyYmw3QmVlUEZZSE9MNGw4VXRUVXRLNHR1UDdtRmwxZnJSUDdJeXRZVWx6THA3MDZxb2l3TFkwM3JKZVNwMzZjd2tBRzVhNTY2ckdlbDFQNDVvSnVidjB0aTY4V0lQQWNUelk1U3hsMTZkTUgyZWhJZG1pYjdDTjVIeklFbVgwSVR2NTRnZUprZ2ZSRU9OUEJCc0w3K3BncEx0Zzc3Z3JYK1hydGl4enk3WnQzVEZQL2dqMWlzTGkzL2EwRmV0d05rSkJjc1JVUEd1Unphclh2SW9rQU9KZWNVU3hLUk1BaTl3VFBCUWZ5SC9zWjFmR3hmekpzRVJBaVJNa25pR01WNDhyYmJseVdQTnI2UmplMk5WZnRTbHJGbGxzRUJDOGkzMm5Ic2ZEZDVUNjdrRVRTV25NY3hnVVdDTnZQalpZSm5VdjFlTWdOZHlRenpQUDZLek5CTCtCOW1CaDZsTWhBNEo1UUltdVhvNDVGcXVUTGVDNHZDRlJiTnk5NVhzS2lYOEV1eHlOYTVZdDZ2YlZYcllyUmE3T0t1MkwyUzZlZkhPUm1pQ3NVQjc0KzF5NEgyWEVua3o4RzRmMXo2VURzc1B1L3NBMnNCaVAwQlEydWx2ZkZFbEVaMlB6ZWt3c2pISnZmUEx1NHYyOGVQbjFScmVVbDJUS0dyRWZaVVNEMkt1bCtsTUxFRXZMdjdDV1F3WStPbUdnQUEiXQo=
"
1678053144001,"START RequestId: b5fbc863-e2d2-4ddf-a281-1ed0c052aba3 Version: $LATEST
"
1678053144008,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.008Z"",""msg"":""Harvest started.""}
"
1678053144009,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWldXL2JPQkQrTDhJKzJqSnZTWDdyOWdBV0NIWUxKSXQ5Q0FxQm9taFhHeDFlVWtycUJQbnZIVkpXNnNSSDA4Uk9VNnhpeEpSRXpueHpjV2JvbTZEU3JjeGxLNFBwVFNCTkhVemQvNm04c3ROU1Zsa3VwNTBkWDJuYmpzbFUwRmhFaU1jNHB2RjAxdFdxTFJvL2RWdzN1ZjdYamhlbXlUdi9kTHhvbXZMOXBhN2JEOFg4YzJ0UEN0c0dvMkJZbEY1cVkrRWJ1UDEyOHVicy9la1p2TlJmdE9yOFcxMWZGcWFwSzFnT005NzhjNXFlZUN4cHp3Zkg0UmVZRDl6YVJqWGxOMkpZakFJNWgxVnI5Sk1RbzVBRXQ2TmdrQklrTm9WSys5dnp1aXZMRVJaUmpEakZqSVl4VGI3ZHNoQ2hHQ0dFUitmbk4wRXRLdzBVVDd2Rm9qR3R6SXF5YUplVFB6Mm15Wm1SdFpWZU9qczUxWE1IM2dMYmN6ekM2eCtDUDMwYTdTUjJ1cEMxVjl2a3JHbGxyMEY3cW5YdEtQRVJHdjZlUUFOMHVZZkdYKzFuYmRaa21IeFltV3J5ZVB0NllSR29qSFBCYVJMN01SRXNZaVFoZk9QTitoZ2hpcEtJQ3M3aUpHYUl3ZjFlZUY2eXM2TFN6OE1wQkNNMEloejVDMjhEdkczSzFvY0FtYUNZOHBnS3drZ2M4V2kvZ21WWi9oU04vUnp4MzNWR09oaS9MOStDNE5wTS9xNHY2dWFxM3ZsOVpQVThCWTlYNlZGQndaWmsyOGJvU2JQUVBiN0p4OGEyYzZQdHhPcFNLKys4MUJPakNBUEJlR09NQkk4NGkvd1lrMFJnbHFEaEdrd0Z0aUl4Qm1ET1VBQURrUjA0MXFWK1FiWjNRcjhDL2krdjhSZm1hRnZaYXBldXZvbTk2TEt5VUdGbklZMnVPVjVQR3NNR1FERjU5Tmhod3BoSENVY01VOFoyd0NocUFGS3JOZVZYbFJ6UFpBMFRscUZxemJYTWFuVzlRR1Y0VjVLRUpyZWhyT1IxVThQMkg2cW1tbkJHeWF0UjRNd25uVTBOYnVYNC9LZHVJRnk5UWdoRnhJdTdhNFBaamRsbHpOUlpQdlhvMDBXaExqWkZXRmZnOThZc3BJUUNMaDVCaGtqZ2l2UTQybGZCdktsbHVXd0xaZnZhWlZzdGhCOVhDKzJsVkxmN0tQMXdnSXdDcTJEZlBuUXhkYVJ3ZTZ6VEhrdXFYellFanFTUUF3UVV5QmxZS1A1VDdaamNhM0J1QWhCUG04dW1NS2t0cmdFOUFYTFFjL2xRU0syTHFpbS85VzFPdS9UU3VUWUNoRzJOVlBxUEhCNkluSWtzNHl5WE9wL3hpQUVDZ1NqTnVMTUIrQjlNbm5lRm16bGpTRk5Oa1lyUURFTko2Sm8xYVlDVHA1TnJybkt1ZUl5d3hJU0puc21nUkQ5RmcrK2lXVUk1emhqS1krV1VMcXRGcWVGbGF6cnRtcitpTVJEaDBQU0ZJZ1l6azlFekFsZkJ2SGxqbGc3ZHNObzlicXBGVS9kZDZFREJvWVVxR2toWEM5ZHhEcDBpK0JtMG1LdXFNcGl1UjZVM1NuaFIxRTQwVlJiOXhuTURuNkEycWY2aXlzNFdsem9kVnFkVlVaYUZkYUlORlBJc0hQSWowSURFdU1xTFRyTmFtL0F6b0Z1Si81U3NPWkNSZVE0aTJpZFNtZmE1MTBmYjg1M28yRzV5aU9pOTV6bHpYV3RUcU4wT1FoNDZ5TkFiZ0J1RVFOZ3NQemFGYzdZZSs1cTA0VUc3ZE85NXptYjlDVTk0OUdNZng4dm8venFnNSsyVzhWbW1Za0hIbXVSa3pQSjhOcFpRYW8yeHpwRkNuTWhNT2tmWUh4d2tISHF1UTNtY1Zna2lMRUs1aUdNNmkvVExibHZPYVl3TnJXNWRsOTUwUCtwZDBVUHZjdmtpU3U1ZGo5cDEzRm9SSDBxcEpFL0VqQW9hRVpMd1RQRFhsd3UrayswUGtoczJqYk5LNVUvT0RTZ2NLUHdQY3dOV09NbDVIZ21sVWN4eTlQcWM2bUVSZlJBdjRoc1ZSbDhEUDZmQ0dDajhVbDdrS2wyNWF1MGVWZTI2UHVCaHVZdjdjdmRLWisvdVorV1lDVXo0dmpSKzBCSWkzOEg5N3NCMnV2WFFkUWlqTmY3dVJ4UmpHaE5NWjdLMHV2L0pJNU5XM3hkd09BaTVlKzJPUU44Mm5mTkMrc1RVdVJsUDJ3TG9Yb3o5S29XSWM3ZmIyNjlodnhNM0toc0FBQT09Il0K
"
1678053144009,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.009Z"",""msg"":""Harvest finished.""}
"
1678053144009,"END RequestId: b5fbc863-e2d2-4ddf-a281-1ed0c052aba3
"
1678053144009,"REPORT RequestId: b5fbc863-e2d2-4ddf-a281-1ed0c052aba3   Duration: 8.36 ms   Billed Duration: 9 ms   Memory Size: 1024 MB    Max Memory Used: 118 MB 
XRAY TraceId: 1-64050f17-d7712cf429df6a1e6cf09c40   SegmentId: 61c6239173afc2c1 Sampled: true   
"
1678053144158,"START RequestId: ccd08014-eccf-4321-bc76-b0143f542429 Version: $LATEST
"
1678053144164,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.164Z"",""msg"":""Harvest started.""}
"
1678053144166,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.165Z"",""msg"":""Harvest finished.""}
"
1678053144166,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWmJXL2JPQXorTDhaOVRCekplckdkYjd1OUFBY01kd1BhdzMwb0JrT1c1YzQzMjhwSmRydTI2SDhmSmNlZDJ5eFoxaWJyaG11RFJKRXBrZzlGaXFSeUV6U3FFNFhvUkxDOENZUnBnNlY3WDRwTHU2eEZreGRpMmR2NXBiTGRQRnB5a3ZBWXNRUW5KRm1XZlN1N1NudlNlYXNMOWErZHI0d3Vlajg3WDJsZHY3NVFiZmVtT3YvUTJiZVY3WUpaTUM3S0xwU3g4QW5TZm52NzR2VDF5U2s4VkorVTdQMVQxVjVVUnJjTkxBZUtGLytjWkcrOUx0a2dCeWZoSjZBSGFaMld1djdDRFBOWklNNWgxWVIvR21JVVJzSHRMQmhSQW1KVHlXejRldGIyZFQzRFBFNFFJNWpTRUtFRUlZU25VNWpUMmRuWlRkQ0tSZ0hIazM2MTBxWVRlVlZYM2RYaVQ2L1Q0dFNJMWdxUHppNU8xTGxUM29MWU0yQTFmVVg0L2Z2WlZtWW5LOUY2c3kxT2RTY0dDOW9UcFZySGljM1ErUDhJSG1ETEhUeis2ajRvTThHd2VMUGVxc1grKyt2QklyQWdvNXloWkJoSE5JNFJKVW1LZUVUVE5HVWJGTk14MEtmZ1pCZ2pRampqZUtlT0h0NXAxYWluS2N0QWFNeThnbXlYeWh0MG16TUlFUlF4Z2doTjRpaUZlVUozVzFuVTlmT1o3WGxzOEtvM3dxbngrOVZMUUsvTTR1LzJZNnN2MjYyZlA4SkdqMUhLMi9YNG1zRXhaVHR0MUVLdjFLRGs0cDIyM2JsUmRtRlZyYVQzWmVJNUVZUjVFcWNiWThSaGk4Z2dEUlBFVWpMdUZTSVJDSTBqSUdXTWJaRThCWHRVUVhmQW5rWGlqN0RqMFdYWVRuVEtKWjh2MEZaOVhsY3k3QzBreFluTHJFVWd3akFtZTQ5ZGNrUXhwaER5VVJUejFIM2Zva3JWZ2pLdG5CaTVhY1M4RkMwUVhJV3lNOWNpYitYMUN0WGhYWkVSbXNLR29oSFh1b1d6UEpTNldUQktvbWMwVytsenhxYmRKakwyR1RzN3hTN3NlUlRSS1BwK1BWd1N5OXdlWmw2amJGWEpqNXRxVGMzd3JURU5HWTlwekNpQ1EwZk40MTFGeFl0VzFGZGRKZTFRVG55dFBNSDdsU2M3T2JYZExrN2Y3ZVd6d0VvNE5nOWQzeHd4WnZiMXcyTWgrNm04K2tnZ254WWpBREd3VUdKbnl2Ry8xMGJjQklCTW1RdGRtY3hXMTZCNEJKeWdzL0hlblZrWEtFdDI2NXVKN3NvRGM4VTY0T3lNa09xUEFpWm9HVkZCc0VqekpDYzRRVElsUktpNDVDbW5CWk1DaU0vN3lsRWlTbE9aQ29Sa0VtSFlCOWNTQ1FPU1BKK2M0NVNMVWhRMGppaExrMEhJYUwrQnBPQzhrSWdqaGdVQm1NN2VvbG5WQ2g1MnBsZXV4YXEwZ2FDRjFpcWttRENHWjArSVJRbDA1OW9BTjkrTCtkVnVXamNyM1E2OTNzakJhUXNWS3JCdVZxNnZHM3N4ek1HY3hicFlDNWJUUVBPYkVuNnNXZ2ROMXRWd2x0ekFLMmhOcGo3SnVyZlZoY3JHMVZsVDFYVmxIYlNSUTVHSFk4NENIcENzMXJuS1dWWXBFMzRBN2Rid0g1UEpSamFpS0FDaWZTU1g1WkFQZmFBOTNZbU83U2FIQ054N25nUHR2WUwyZmF1RHNQU2hnd3lsTmtySFA5RE5oQ0RCWEwzVGxmTzZBY1FFZG5qUXB0aTdvTnU4NFVJbFBQb3RpNU5sMUg4OThQTWJLR1VCK0RHZEt5bkxPWGdQbnVjeTV2TWM1a2pKS0p6Y2FURDdScFJFNFlPZWhoL0tCY3NVNTRtUTBDOFNhSUZVL0dQUE1lZEZ4b1pXZGE0bDF2MTN1aHVuRDkzTjVRN0N5RjVIajZQM3hBY3lKYU9scENvdkkwUWtTUlAwODZXRWIrVDdnNlFJc3JFbDYyVCs2QlNCd3BIRC96QkZrRWpHRVV0UlFTUVdvaXgrUHFkNldCb2Z4SXZ3UnFHeHJudWZVR2lNSEg0cEwzSUZyMWczYlhzVnZSaHRWcjE0cUhvdlZmN3FmbktPaWV2aHQyYno1TUNWUkxGRit0MmQ2SEx6Vm5PTW9ZbHc5M09GTWRvRXkxTFVWZzAvTHVUQ3F2dm94Z3VLdThmdWR2R2w3cDBMa2tjV2JKdkI5TFhvdVJkZ3Ywb040bnp0OXZZemxNRzBQcFFhQUFBPSJdCg==
"
1678053144166,"END RequestId: ccd08014-eccf-4321-bc76-b0143f542429
"
1678053144166,"REPORT RequestId: ccd08014-eccf-4321-bc76-b0143f542429   Duration: 8.81 ms   Billed Duration: 9 ms   Memory Size: 1024 MB    Max Memory Used: 118 MB 
XRAY TraceId: 1-64050f18-50d35116d8e9ac54e61f0b57   SegmentId: 121706c2360d5123 Sampled: true   
"
1678053144343,"START RequestId: be481005-08eb-4337-999f-88dbbae9db0e Version: $LATEST
"
1678053144349,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.349Z"",""msg"":""Harvest started.""}
"
1678053144351,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.350Z"",""msg"":""Harvest finished.""}
"
1678053144351,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWldXL2JPQkQrTDhJKzJqUlBrZlJidHdkMmdYYTNRRkxzUTFBWWxFUW4yc3FTbDVSeUl2KzlROGxLSFR0MlhjZEpVNnh5bUpLb2J3NStvNW1oYjZLWnJVMW1haE9OYnlManltZ2Mvby9OaFI4WFpwWmtadHo0NFlYMTlaQ09ZNlppaVlVaWlxbnh0Q25UT3EvYXFjT3l5dXkvZmpoM1ZkYTBWNGZ6cWlyZW50dXlmcGVmbnRYK2ZlN3JhQkQxRDAzT3JmUHdDZEorZS8vcStPM1JNZHkwbHpadDJydTJQTTlkVmM3Z2Naang2cCtqeWZ0V2wwa25oeWgwQ2ZOQldsMmxWZkVOak1TRHlKekNVMHY0R2hHTWFIUTdpSG9yd1dLWHA1UHU5S1JzaW1KQVlxbXdZSVJ6UkdLK2ZNcTRIcHljM0VTbG1WbEErMkJubGJzYWZUeTc4bmxxQ29BOUlRTUpBaVRqVElydkQ3bG1DakVSWTZKd0hITWNmLzQ4V0lQL3c1cjU2Sk8zV1lkUFlrUklETjRIeHdzZFVQYTdRb1ZHa3NlWXhveGpKU2tXRzRWL01KZWRiSWFSNGpJV01SVTdqYlVnU0VpcGRFeUlBbCtTalNMZU9Xc1g5bkVrR2JuekNLRHNkNFVTaVRDbVNqTXFxVlNTc1llRS8xV1ZLLzVsR25FcXBDTEJsaDNHUkFnZ2hxYVVTaUtwRXVUK0loNDE4M25sYXBQa1JWNEhlWUd6bzJOblNtOWE5dnZSa1QwTjVQWUwrNWQvS2RrR2RqUTNaUnRXbytPcU5sMkUrU05yeTRBa0Jyai8yUU1EWW0wTHh0LzFtWFZMTm96ZUxVSjV0SHY4dDhaaVdDSXVPV094YXNjVVM4WENta21CdzhIV1ppeVBZVDdsV2d1bXRXS2NjcnpDNFZVMVd3dVA4NWw5bkw1YWdyeFd4MjA2cjh4YVBZZm51R3I1cVlpTUtXVlNiWGV4S1lxZjZyUG5kc0NieHBtZ3dPOVhyOEYwNjBhZnlpOWxkVkZ1L0h3bUIrMmpWK3ZVWjFFTzBwaXZLMmRIMWR4MmVvNCtWcjQrZGRhUHZDMXMyaEs1RXdKU09lZHliWXhqRXRPNFU1VlFxam5oQytHWWFrMklnRDhpTUlGenZrSDZzczFQTHV6T3dKOG05Ymw4K2l4eWZHMXFHeExTTnhQblRWTGtLV284RkZKTE5GcUl3VnhKcUIvNjQxQlh3d0JyRFpsVllxa1ZEeVpzVURrdlFla3lYVnFVMmN3TXA2YUVDVmNvcmQyMVNjcjBlbzRMZEZmQUlwZDVaR2JtdWlvaENhQzBtbzBFWi9RbnUzamFKcHgxSHkvSjJXVWNmQ2NJNFVMQjI0WHBPSnovc0M0aEMwN0NtazlhclNielBQMnlydHF5Tzc0M1praHFLZUZ0RDhvU081VGJDcE5YcFNtdTZqejFYVW55VUlsRGRpdHh0aUtWOVRha0g0NktRZVJUZVBVZXVrWjZBVEcySzJlZnlnTXZMZ0tleU5ESHhST1lHSGtvNlNjMjROOXJhMjhpc015Njh5cDNFNTlmZytLQkRkQnB0NUV3OFNHb3h1SzJiWERycTlhdzBCeUFuYlV6cWYwemd3c2tUVmlTSlBDaUpHUnFzMWhQbVUybU5EVmE0MFFhQzVOUG16ek10SHlxTXkyWmxSblhtRTVEaTI0Y1NHcHhCTFNkTkk2eHBhSDBNZE5PU08rL2Rrb0toYUhsR2NsRUp1UHBGQWQvbTltOGdENXRYTHZHaHBZL3J4d0VPTFQ2U090WVlUVjRSTnltTU84VVdrSjRPdXVmRHBlcjJid3F1NzJISGlGb0N4VXhRTS9tWVoraDN4OWdYQXlpYkZFZlJ1T0h3NjVkSHZRbEwxc2ppN3g3QTkzQWIxUzZpYjFNaThibjUzYlM0MHhtZVZIa1BoalpZWUdJQlBVWkR6QWcxUzB5WGZDeHRRNmRnWjRMUit5VEIzc1lrMlZnck44VFpkeGwwemJrSGsrbnB5Yk1JVUw0SG9kT2JXbGRubTZtQ2wrbFNsL25BdzBRQUx1cmoxVWVhTmZwdm1RdE9tZ1gzakl2ckZtM3c0ZWVmTnN2eUhMMnZ3YncyblZMTEZkUUxJa2hWallaZ2cva0VETFNkS2hVbGlURzZpekJnUWpiZzRPaTFUN3FVTXhMcllvTjNNc2d0U1F5ZmVZWFdTQ1A4OGpiT3ZUZ1ZmT2pMTk9yTElORFVnS0Y2cmRqcDdkUGVCWWVQSmhiQ1dlS3BWeFl3ekpPZFBieThzTjNrdjlCOG9WYVc1NUZadDg3UzJEVUkvd1BzMFNjY29KdHl1UTBzMFpqL1BKSXRWb3JINFJGY3EzcVdCVENqNmcxZW9SZmlrV2graldMYm0rbkNqZzA2YXNsTU9sSzRBdWJ2TG5uVkdqcE5aVGEyeEw2UVl1SmJJUDB1dzNaOGVxMmFoOUJTNkxETjJuT1ZTNGFUMDNoYmZlOVYySzh2Vzlidjc5eGR6dnNiYjZ1bWtCQXRtZmVYQStsaDJMblhuajlLdFZJWU5ydDdWZWxSY0YyTHgwQUFBPT0iXQo=
"
1678053144351,"END RequestId: be481005-08eb-4337-999f-88dbbae9db0e
"
1678053144351,"REPORT RequestId: be481005-08eb-4337-999f-88dbbae9db0e   Duration: 7.80 ms   Billed Duration: 8 ms   Memory Size: 1024 MB    Max Memory Used: 119 MB 
XRAY TraceId: 1-64050f18-1ecbc24f5e5c74cb2a973ab0   SegmentId: 51e7739403870e88 Sampled: true   
"
1678053144527,"START RequestId: 91d62183-7082-4017-8d61-9535a0270ec9 Version: $LATEST
"
1678053144534,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.534Z"",""msg"":""Harvest started.""}
"
1678053144535,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.535Z"",""msg"":""Harvest finished.""}
"
1678053144535,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWldXL2JSaEQrTDBRZkpXb1A3cVczTkFkUUlHZ0QyRVVmaklCWWtpdUhEUTkxbDdSakcvN3ZuVjJLRG4zSVVXeXBUbEJia0NoeStNMngzK3pNVUZkUmJUcGQ2RTVIeTZ0STJ5WmErdmVsUG5mTFN0ZFpvWmU5bTU4YjE4M0prbFBKQldJU1N5cVhxNzdKdTdJTm92T21MY3pmYnI2MmJkR0hzL04xMjFadnowelR2U3RQUDNYdWZlbTZhQmFOTjZWbnhqcjRCRzIvdkg5MS9QYm9HQzZhTHlidncxWFRuSlcyYldxNEhTUmUvWFdVdmcrMnBJTWVMT012SUEvYXVqWnZxNjlnbU04aWZRcDNUZkJWakZGTW91dFpOSG9KSHRzeVQ0ZXZKMDFmVlRQTWhVU000aVNKYWFLbVh4bE5aaWNuVjFHamF3Tm9SLzE2M2RwT1oyVlZkaGVMMzRNOWkyT3JHNmVEWjI1eFpFNjk0UTVVbnVBWm5yNEkvdmh4dGhYc2FLMmJFTExGY2R2cElYcnV5SmpHSTdFWkd2K2ZnQUZ4ZkFUamorNlRzUk1mRnU4Mnk3VFlmVzJEc3loR2lCRXA1WEJJT0JNSkp6U2hndDI5TmprRVFhRUVaaGppdzVHQ1AvR29lY0d6NDdJMno3T1RDMHlVRE1hQkNkc3Nma2pxd1pNSVVUQ2ZjcElvUkNYQjdQRVE2NnA2a1ppOVZBRGU5Rlo3UTM2OWVBMnVHN3Y0cy9uY3RPZk4xczhEQitncDlvU2dIdFFvMkpKYzExcXphTmRtc0cveG9YWGRxVFZ1NFV4bDhrQmdHc0FvUWdCMTl4QnhuakF4U0dCQ1dNTHBSaldpV0dIS3dSN0dWYkpGNzlUTEE2cTVjZW9GOUIwK2dnZlc0RHJkR1Y5aXZycTE3ck9xek9QZVFkbWJFR1ZnSGNaWVlKenNmT3d6bWlSQ0paUkxxUlRmWWtiWmdDRk5QZ2x1WGV2NVNqY2djQkhubmIzVVdaTmZybEVWMzdRUXNTMWNyR3Q5MlRhd1pjZDVXeTlZUXNtTEJXd1ZDc1A5aUUxMDdITHNJOFlRRmt3eERsRlQzMjJIcjFTcFg3MDBXSlN1eS96emZiT21ZZmpXY1JJbkdEWWxJZ2dseXN6RlkwM0RxMFpYRjEyWnU2RmRlS2o5d0x1MUg0OGlOZDFqU04vTjcxbmtjdGdtOTkyL0hDaGJkdVhnb2J6Nm9SaDlJQ2VmbHgvZ1l1U2dmVTZOeDc4MUhseEY0Sm14WjIxcFUxZGVndUVFa0dCaUNjeE9uVStTSmJzT2cwSjNFUnp6alRqNDJWbWRtOThLT0pGcFRsYk1vSXhtZ2hFdUVaZXM0Q2pURW9zRVpRS0VUL3ZTUzJLMmdqMVJaVUxCMjhva2Z0VFJGalFGSENQeVZaYUpERkdCaXFRZ2c1SXhma0ZFMEZXQ1ZTWUo0aGpsa3Z0NDYzcGRHYmpZMmQ3NDBhbHNMU1FzakV5eG9vSXlPbnRHSHVZZ2Q5cGFRQXN6VnJqYm4yN3JkZHNNTTl5STRLMkZMaFNnNjdXZjE4WTVpeEVGQTlxbUo0dVcweVFMaXhKL0xodnZXbDZWd3o1eUJhK29zYW41a2xlOUs4OU1PdDZkMW1WVmxjNjdOaUlVV1R4V0s4Q0FNcldwVWo2eXh0ajRFMWkzY2Y4cE5XeUUwVVVCTHJvbm9peUhTaGdTN2Zra09qUk45cEc0dDVnRFk3dUJzWHc3UWVSZGdtd2FhMkJCRExqMjRrTmJlcTROcGsrY2pmYzY1Z2JpK1NVYkhvL0VCMzltNG5WWjgwOFBlR0haRkM0NEFiaTVRSkxNRTlpazU3TGdlSzRZWlJvR0RHUnlGYzIra1Jza25rd3MrK0ljNGhtVmlSU2FLbzAxME9nLzNiZzhiYXlMbmVuOG5OdjIzOGt2U3UveUt4UUx5WGZhYTd4OEVONVRLRm5CRENsb1RoS1FVRlQrZURYZ0d3VitIelhoZ1NYWlZPOG4xd1FVandqL3c1cUFvVUhWU2hZRVlTTlJrdjE0cExyYkMrK0ZSZmhlWjdGcGRKL1JXWXdJUHhXTGZJZXJOeFBhVGwwdVJ2ZmJYRHkwdWVjbWUzTzdHak9JdEcrMnRwZnZ2YllPeFJidE53ODZsM2VmVk03R0ZKcm85ajg3V052YWFMblNsVFBEandTWmR1YTJjNXRuRVRkWC9UUEQxMjN2Q1VpZmxvc1BwTkpEdVhNcnZYNlc1c016N2ZyNlg3TlBBRjViR2dBQSJdCg==
"
1678053144535,"END RequestId: 91d62183-7082-4017-8d61-9535a0270ec9
"
1678053144535,"REPORT RequestId: 91d62183-7082-4017-8d61-9535a0270ec9   Duration: 8.13 ms   Billed Duration: 9 ms   Memory Size: 1024 MB    Max Memory Used: 119 MB 
XRAY TraceId: 1-64050f18-5b33f0137f02ea6f0b89e673   SegmentId: 6c9b2a2d45a9bd47 Sampled: true   
"
1678053144650,"START RequestId: 5106f6f7-6032-48d6-883a-4f8d0da3b315 Version: $LATEST
"
1678053144657,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.657Z"",""msg"":""Harvest started.""}
"
1678053144658,"{""v"":0,""level"":30,""name"":""newrelic"",""hostname"":""169.254.194.189"",""pid"":17,""time"":""2023-03-05T21:52:24.658Z"",""msg"":""Harvest finished.""}
"
1678053144658,"[NR_EXT] Agent telemetry bytes: WzEsIk5SX0xBTUJEQV9NT05JVE9SSU5HIiwiSDRzSUFBQUFBQUFBQStWWlcyL2JOaFQrTDhJZWJabVVSSXIwVzlZTE1DRFlDaVRESG9KQ29DZ3ExU3FKSGlubGl2ejNIbEpXNnRpSjZ5WjIwMktPRVZNbWRXNzhEczkzNU51Z1VaMG9SQ2VDK1cwZ1RCdk0zZis1dUxUeldqUjVJZWE5blY0cTIwMmpPWTBaVFJGaG1NVnNYdmF0N0NydGwwNWJYYWgvN1hSaGROSDdiNmNMcmV0M0Y2cnQzbGZubnpwN1hOa3VtQVRqVGRtRk1oWStRZHR2eDBlbjcwNU9ZVkpkS2RuN1dkVmVWRWEzRGR3T0s0NytPY21PdlMzWm9BZXo4QXJXZzdaT1MxMS9GWWJwSkJEbmNOZUtmQjVpRkViQjNTUVl2UVNQVFNXejRmS3M3ZXQ2Z21uS0VJbHhrb1FrVGxZdktVa25aMmUzUVNzYUJkSk8rc1ZDbTA3a1ZWMTExN00vdlQyelV5TmFLN3huZG5haXpwM2hGbFNlNFFsZWZVZjQ0OGZKazhKT0ZxTDFJWnVkNms0TTBiTW5TclZPRXBtZzhlOFpNaUNPVzJUODFYMVNac1dIMmZ2bE5zMTIzMXZ2TEFvUlNqaUoweVQxNHdoSE1hV01wV1JqYW5VTUN4TVNjNHA0bk1BRjQ1eHV0Yys3ZGxvMTZrV0dFb3dqbXFEUnVEVmIxMmJYTG1FNXhURkhuS2FBbFFpbGxHMlBxS2pyMXduUmovSDNiVytFVS92NzlSdndWSm5aMyszblZsKzJUMzRlT2g3UE1jakg4TEJXd1pGak8yM1VUQy9VWU9Ec2c3YmR1VkYyWmxXdHBNZG5QT2prRENmcDVoakJNY3p4TU1hWVkweVRwWElVWXhLbFBBRWpJb2FmMEx6cTZFRVYzVHYyS2hwL1JCd1Byc04yb2xPdW1IeDFiZEhuZFNYRDNrS0JXNEVNM2xDeHl4aGVPTUVBYTBqcWhEMEoxcW9GUTFxNUV1Q21FZE5TdExEZ09wU2R1UkY1SzI4V3FBN3Z5VUpvQ2h1S1J0em9GczdtVU9wbVJwSTRlc1dRbGI0R1BCSXpBaWNiM1huc1loWkhKTVVSaEl4aDZxNi8yeFpYbURLM2g1bTNLbHRVOHZPbWFhdWgrTlk0Q1JHTEVHY1JKaFRPSGM2Sm1xYmJtTUpSSytycnJwSjI0QWlQY1E2OEcrZllLcW50dGtuNmJxaFBBaXZoN053M2FUbFE0dXdLeG9ONTliTkIrMENPN2lGUndNL0FBbm5PbEZQeW9EbTREY0E5WlM1MFpUSmIzWUQxRVlpRGZzVkRQTE11Vytia3pyY0ozYlgzenRGd2NMWXpRcW8vQ3ZpQ3hES1hpc2NSV0ZFcUxCUmpTa1Zsd1VzV1JUeFBZZkY1WDdtVkdFbUNoT1JBdUpJaXlpUFg2QWdEbXJ3Y0tpWG5LU3Z6UkpXa1RPTkJ5UmhFdjBTV3BjeEprZENDU3B3UTRvSXVta1d0WUxJenZYS05VNlVOWkM0MFRLR0xDckNhRnlTa2hIWG4yb0EwMzJINXU5M1h1bG5vZHVqZ1Jnbk9XdUNrSUxwWnVHNXQ3TElvaWFBOVcxSzJZTDZhYlg1VHdzOVY2MTJycStGQXVZVjMwSnBNWGNtNnQ5V0Z5c2E3czZhcTY4bzYxMFlKUlI2T0ZReGtRT2xhVmk0WFdhVk0rQW1zVzdyL25MbzJpaEZGQVM3YVowcVpEOVhSWjl2TFFYUm9tT3dqZXg4Z0I1cDJCVTM1MHdEQjZ3QVplVGZBSUFUQjV2cURyaHpZQnR0WHZBMzMydVY2NUxrOUc1Nk9oQWQvWk9KMEdmVmZEL0lHRUdCRVMxcW1VNHJpYUpxd2drNFppOFUwS1ZtQkNoSG53SktDeVRlU0l3cnZHNXA5UVE1ajF5WnhWZ2ljUzU3ekgzdHVPZFFZRzFyVnVhWlg5OThMTDdvT0wzaWxDZWM3SFRWdXZWKzhwMUJLVmlLaGlJeHppdE5TeXArdkJIeWp5TytsSkpDTkxWbFc4R2VYQkJTT0V2NkhKUUd3SWdxWnhpbE5jYW9RK2ZsQXRjNko5NEtpWkpOWURJVDNKY1JpS2VHWFFwRWp1R0xacWUxRWNqSGFaTGw0WUxtWEtuLzdJS2drU3FCemlMZFY3NzB5aCtJSjdmZFBQZWRyRHpISEJGclI3SDV4TUVhYllGNksycXJoOTRGY1dQWFF0ZkhaeFAyMGU2RDRSdmNPZi9FenovZk5USG9zZFI1azE2L0NQQnpRN3U2K0FDMXhlWkJYR2dBQSJdCg==
"
1678053144658,"END RequestId: 5106f6f7-6032-48d6-883a-4f8d0da3b315
"
1678053144658,"REPORT RequestId: 5106f6f7-6032-48d6-883a-4f8d0da3b315   Duration: 7.97 ms   Billed Duration: 8 ms   Memory Size: 1024 MB    Max Memory Used: 119 MB 
XRAY TraceId: 1-64050f18-43f5bf1fcb2dbf33ec6ab3d8   SegmentId: 6084f21f5d603467 Sampled: true   
"
1678053528202,"[NR_EXT] New Relic Extension shutting down after 9 events
"
1678053528403,"[NR_EXT] Log server terminating: http: Server closed
"
1678053528403,"[NR_EXT] Aggressive harvest yielded 7 invocations
"
1678053528618,"[NR_EXT] Sent 1/1 New Relic payload batches with 14 log events successfully in 214.902ms (213ms to transmit 10.2kB).
"
1678053528618,"[NR_EXT] Waiting for background tasks to complete
"
1678053528618,"[NR_EXT] Extension shutdown after 389923ms
"
lucasklaassen commented 1 year ago

As you can see there are no errors in the lambda logs, but the response coming from API gateway is a 502 due to the body not being passed through.

I've also tried to remove the license key from the ENV and letting the API graph it using my API key so that warning isn't the reason.

lucasklaassen commented 1 year ago

I've attempted to get this to work with both API Gateway v1 and API Gateway v2 and both don't handle passing through the body through to the response.

mrickard commented 1 year ago

@lucasklaassen It's odd that the body's being set to null. The New Relic Agent shouldn't be resetting the value you're using for the response body. I'd suggest opening a ticket with New Relic's Global Technical Support--they can take a look at function code, account-related values, etc. How are you initializing and assigning responseObj? Are you referencing New Relic agent methods in that process?

This is more of an issue for the newrelic-lambda-layers repo and its Node wrapper, I think. The plugin for Serverless is mainly an aid for deployment configuration, and your function's been deployed successfully (even though it's running into this problem with the response body).

lucasklaassen commented 1 year ago

I'm able to successfully wrap my lambda functions manually by including the lambda layer and using the newrelic.setLambdaHelper workflow which is why I've made the comment here as when I use this library to do it all for me, the response object returns null.

I'm using async and 

return {
  statusCode: 200,
  body: "response"
}

Doing the manual instrumentation flow adds a lot of extra latency to all my lambdas. It seems like it's hitting some sort of timeout. It does return but all my lambda workflows slow down, 5+ seconds for api gateway lambdas and then any background scheduled lambdas hit whatever timeout they are set to, 300 seconds in my case. (looks like the same issue as this: https://github.com/newrelic/newrelic-lambda-layers/issues/140) Normally this happens when the method is waiting for the event loop to have nothing in it but I do use middyjs in my project and have a middleware which sets it to return even if the event loop isn't empty.

lucasklaassen commented 1 year ago

So even with a successful lambda execution when using the pg library, if I make a postgres query like so:

const client = new Client(configurations);
await client.connect();
const results = await client.query(text, values);

the response breaks between lambda and api gateway for some reason when using the serverless-newrelic-lambda-layers plugin.

lucasklaassen commented 1 year ago

So here's a lightweight repo which can be used to reproduce this issue: https://github.com/lucasklaassen/newrelic-lambda-layers-bug

After you deploy and hit your lambda through the API endpoint it will return fast and function properly but when you rename your lambda handler to newrelic-lambda-wrapper.handler in the AWS console you will experience timeouts and latency issues.

I've tried using the serverless-newrelic-lambda-layers plugin as well as manually instrumenting the lambda function and I either run into timeout/latency errors or I run into a response not returning.

It doesn't seem to be related directly to me using middyjs or pg it seems to be related to how the arn:aws:lambda:us-west-2:451483290750:layer:NewRelicNodeJS18X:16 is working.

I've spent the past 3 days debugging this, trying every single combination of deploying using all the documentation you have publicly available and searching through all of github to see how other projects are using this integration.

jordigh commented 1 year ago

Try this simplified serverless.yaml instead:

service: aws-node-http-api-project
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs18.x
  region: us-west-2

functions:
  api:
    handler: index.handler
    events:
      - httpApi:
          path: /
          method: get

plugins:
  - serverless-plugin-typescript
  - serverless-newrelic-lambda-layers

custom:
  newRelic:
    accountId: # Your New Relic account ID here, e.g. 123567
    apiKey: # Your New Relic user API key here, e.g. NRAK-2I0ABCDEFGHI1234689

and also simplify your newrelic.ts to not manually instrument:

const main = async (event: any) => {
  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: "Go Serverless v3.0! Your function executed successfully!",
        input: event,
      },
      null,
      2
    ),
  };
}

module.exports.handler = main;

This should be using automatic instrumentation via the plugin instead of manual instrumentation.

lucasklaassen commented 1 year ago

@jordigh I'll try this tonight. How do I set my APP Name using this flow which will display in the newrelic APM ui?

jordigh commented 1 year ago

The service name ends up being used as the New Relic app name.

lucasklaassen commented 1 year ago

@jordigh Using this flow I'm not seeing my app show up in the new relic dashboard within the APM section.

I am seeing Span's get logged in the product though, so it is sending data. Any advice on getting this project to show up in entities or apm?

lucasklaassen commented 1 year ago

@jordigh I tried the simple example you provided at it worked except for the fact that the app name isn't displaying within the APM, can only see some data through querying spans with NRQL.

However I'm able to recreate the issue when using middy and pg.

I've updated my example to still be lightweight but showcase the problem: https://github.com/lucasklaassen/newrelic-lambda-layers-bug/blob/main/index.ts

If I remove middy it works as expected AND if I remove pg and leave in middy it works as expected. So it's only when they both are included the api gateway response returns null and when I remove the serverless-newrelic-lambda-layers plugin, both pg and middy work as expected so it definitely only happens when the layer wraps the method.

service: aws-node-http-api-project
frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs18.x
  region: us-west-2
  profile: personal
  stage: production

functions:
  api:
    handler: index.handler
    environment:
      POSTGRES_USERNAME: ''
      POSTGRES_PASSWORD: ''
      POSTGRES_HOST: ''
      POSTGRES_PORT: 5432
      POSTGRES_DATABASE: ''
    events:
      - httpApi:
          path: /
          method: get

plugins:
  - serverless-plugin-typescript
  - serverless-newrelic-lambda-layers

custom:
  newRelic:
    accountId: 123
    apiKey: ''
import { Client } from "pg";
const middy = require('middy');

const logging = () => ({
  before: (handler: any, next: Function) => {
    console.log('before', handler);

    return next();
  },
  after: (handler: any, next: Function) => {
    console.log('after', handler);

    return next();
  },
  onError: (handler: any, next: Function) => {
    console.log('onError', handler);

    return next();
  },
});

const main = async (event: any, context: any) => {
  context.callbackWaitsForEmptyEventLoop = false;
  let results;
  const text = `
    SELECT 1
  `
  const values: any[] = [];
  try {
    const configurations: any = {
      user: process.env.POSTGRES_USERNAME,
      host: process.env.POSTGRES_HOST,
      port: process.env.POSTGRES_PORT,
      database: process.env.POSTGRES_DATABASE,
      password: process.env.POSTGRES_PASSWORD,
      connectionTimeoutMillis: 5000,
      idleTimeoutMillis: 30000,
      max: 10,
      ssl: {
        rejectUnauthorized: true,
      },
      allowExitOnIdle: true,
    };
    const client = new Client(configurations);
    await client.connect();
    results = await client.query(text, values);
  } catch(error) {
    console.log('ERROR', error);
  }

  console.log('RESULTS', results);

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: "Go Serverless v3.0! Your function executed successfully!",
        input: event,
      },
      null,
      2
    ),
  };
};

const mainHandler = middy(main)
  .use(logging())

module.exports.handler = mainHandler;
edwinwong90 commented 1 year ago

@lucasklaassen I have the same issue like you. It seem like middy and newrelic-lambda-layers cannot be coexist in the first place. In my case because my middy version is quite old so I decide to latest version, and after upgrade it works now. Hope that help

lucasklaassen commented 1 year ago

@edwinwong90 after upgrading middy you've seen no issues?

I remember trying that in the past but I think I may have had too many different configurations going on that attempt.

Are you using the serverless plugin to auto wrap your functions?

edwinwong90 commented 1 year ago

@lucasklaassen yes no issue. If you wonder my library version here is:

I only have middy wrap to my functions only.

joeythomaschaske commented 1 year ago

I'm wondering if this issue I reported in a separate repo is related to this

https://github.com/newrelic/newrelic-lambda-cli/issues/234

I'm getting 502 errors when I have the layers installed with aws-severless-express

mrickard commented 1 year ago

@joeythomaschaske I'd recommend opening a support ticket--they can guide you in collecting environment and dependency information (like the version of aws-serverless-express...which to the best of my knowledge is an older project that's been succeeded by serverless-express.)

mrickard commented 1 year ago

@lucasklaassen Our apologies for the long delay in investigating. Thank you for the repro case. This is confirmed to be a conflict with an old version of middy; updating to a recent version resolves it.