Escape characters are being added to NR_LAMBDA_MONITORING payload

keegoid-nr commented 2 years ago

Issue

Escape characters have to be removed before the payload can be base64 decoded.

Is this adding escape chars to NR_LAMBDA_MONITORING payload? https://github.com/newrelic/newrelic-lambda-tracer-java/blob/9db7d75b5cbcd070db4d6a321da464931ae166ef/src/main/java/com/newrelic/opentracing/util/ProtocolUtil.java#L107

Log output from function

START RequestId: 06e322c1-53a5-4cfe-a25b-224519bfae80 Version: $LATEST
[NR_EXT] New Relic Lambda Extension starting up
[NR_EXT] Registration response: {"functionName":"kmullaney-java-walkthrough","functionVersion":"$LATEST","handler":"com.newrelic.java.HandlerWrapper::handleRequest"}
[NR_EXT] Extension telemetry processing disabled
[NR_EXT] Starting no-op mode, no telemetry will be sent
EXTENSION   Name: newrelic-lambda-extension State: Ready    Events: [SHUTDOWN,INVOKE]
Hello, Keegoid
[2,"NR_LAMBDA_MONITORING",{"agent_version":"2.2.1","protocol_version":16,"agent_language":"java","execution_environment":"AWS_Lambda_java11","arn":"arn:aws:lambda:us-west-2:[REDACTED]:function:kmullaney-java-walkthrough","metadata_version":2},"H4sIAJJfdGIAA91UTW\/bMAy951cEPsepJX\/E9q1YO6ADhg1tsUtQBLTMJFpt2ZPlBG6R\/z7Jide5TZtkH8CwiySQTyT1+MTHwXBoVSWIGa5QqFkKCqx4ONXm4VDUWTZqT4\/tqqEtqppViELD6KizS6xQrgouZxV\/QONqPZstYLqDdftTwPZyWktQvDARnbFLIsfxRz\/7lQSGV6l2Wz5JWIiMsCAJ5y5LrR5QQG5yW0sQaYbX+K3GSvUhi5q3gdIkoknge75PgVE676NUU7aBbjQzfQ8DhYtCNsa7QIGSsz6glLyQXBkAGQcTxyNBz19BXmZoalCyxn75cqzZlc3nggu1D6CJEBUww9WOjtSLAt+dsCRxCYbhs1fwXL9f5zO1BD6ZRC4JItfzfoA2o\/0dYXWlivxcKcmTWrVUPDN9gazGfjpYV+MM8iSFMUjTTEtvsbbGW2tcV\/ZaF2TTeHp9eXH+7vby4i6e16J9UHyfa7mBwMb+Ciuw15Ddq6Us6sXy1TSsyNIbBXIvWQYntxLYkuUE6FLKiO274Nsem6MN1E9sSj2fRMkcMHSsvdRsdse70QlCdhxnEoa\/J+SPzQfNxUsVdipOHBd1V2lIkMIkgH9BxSVIreKrY37Zn5BzEDgH5fzAzR3rAcoD\/R10a9tpCwRkjeLsl4YjeX04khOG49Ga6bp9+8TqaT09NIY7UX5SS5Rn73f\/9uzYf9tp9u0mvyGsvz\/ODgb6z0bcS9kPNoPvgcM7eBMIAAA="]
END RequestId: 06e322c1-53a5-4cfe-a25b-224519bfae80
REPORT RequestId: 06e322c1-53a5-4cfe-a25b-224519bfae80  Duration: 415.06 ms Billed Duration: 416 ms Memory Size: 512 MB Max Memory Used: 120 MB Init Duration: 1376.22 ms

Reproduce the error

Install a function with the NewRelicJava11 layer.

SAM config:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: 'A simple Lambda, with New Relic telemetry'
Resources:
kmullaneyjavawalkthrough:
    Type: 'AWS::Serverless::Function'
    Properties:
    Handler: 'com.newrelic.java.HandlerWrapper::handleRequest'
    Runtime: java11
    CodeUri: .
    Description: 'A simple Lambda, with New Relic telemetry'
    MemorySize: 512
    Timeout: 3
    Role: >-
        arn:aws:iam::[REDACTED]:role/kmullaney-java-walkthroug-kmullaneyJavaWalkthrough-1OZ5O1JNDK4E2
    Environment:
        Variables:
        NEW_RELIC_ACCOUNT_ID: '[REDACTED]'
        NEW_RELIC_DEBUG: 'true'
        NEW_RELIC_EXTENSION_LOG_LEVEL: DEBUG
        NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS: 'true'
        NEW_RELIC_LAMBDA_EXTENSION_ENABLED: 'false'
        NEW_RELIC_LAMBDA_HANDLER: 'com.newrelic.lambda.example.App::handleRequest'
        NEW_RELIC_LICENSE_KEY_SECRET: KMULLANEY_LICENSE_KEY
    Tags:
        'lambda:createdBy': SAM
    Layers:
        - 'arn:aws:lambda:us-west-2:451483290750:layer:NewRelicJava11:18'

Copy NR_LAMBDA_MONITORING payload from CloudWatch logs or the function's log output on the Test page and attempt to decode it.

H4sIAJJfdGIAA91UTW\/bMAy951cEPsepJX\/E9q1YO6ADhg1tsUtQBLTMJFpt2ZPlBG6R\/z7Jide5TZtkH8CwiySQTyT1+MTHwXBoVSWIGa5QqFkKCqx4ONXm4VDUWTZqT4\/tqqEtqppViELD6KizS6xQrgouZxV\/QONqPZstYLqDdftTwPZyWktQvDARnbFLIsfxRz\/7lQSGV6l2Wz5JWIiMsCAJ5y5LrR5QQG5yW0sQaYbX+K3GSvUhi5q3gdIkoknge75PgVE676NUU7aBbjQzfQ8DhYtCNsa7QIGSsz6glLyQXBkAGQcTxyNBz19BXmZoalCyxn75cqzZlc3nggu1D6CJEBUww9WOjtSLAt+dsCRxCYbhs1fwXL9f5zO1BD6ZRC4JItfzfoA2o\/0dYXWlivxcKcmTWrVUPDN9gazGfjpYV+MM8iSFMUjTTEtvsbbGW2tcV\/ZaF2TTeHp9eXH+7vby4i6e16J9UHyfa7mBwMb+Ciuw15Ddq6Us6sXy1TSsyNIbBXIvWQYntxLYkuUE6FLKiO274Nsem6MN1E9sSj2fRMkcMHSsvdRsdse70QlCdhxnEoa\/J+SPzQfNxUsVdipOHBd1V2lIkMIkgH9BxSVIreKrY37Zn5BzEDgH5fzAzR3rAcoD\/R10a9tpCwRkjeLsl4YjeX04khOG49Ga6bp9+8TqaT09NIY7UX5SS5Rn73f\/9uzYf9tp9u0mvyGsvz\/ODgb6z0bcS9kPNoPvgcM7eBMIAAA=

$ pbpaste | base64 -d | gunzip | jq .
base64: invalid input

gzip: stdin: unexpected end of file

After trimming escape chars from the payload, it can be decoded.

$ pbpaste | tr -d '\\' | base64 -d | gunzip | jq .
{
"span_event_data": [
    null,
    {
    "events_seen": 2,
    "reservoir_size": 2
    },
    [
    [
        {
        "duration": 0.319005,
        "traceId": "51bc8ec1c6b8f3cd",
        "name": "handleRequest",
        "guid": "db92b654552ac22f",
        "type": "Span",
        "category": "generic",
        "priority": 1.670416,
        "sampled": true,
        "nr.entryPoint": true,
        "transactionId": "5d496537cbb31e88",
        "timestamp": 1651793169344
        },

Expected behavior

It's worth noting that this only happens for the New Relic OpenTracing Java Agent. Both NodeJS and Python agents omit escape characters from the NR_LAMBDA_MONITORING payload.

(Migrate to Jira)

kford-newrelic commented 2 years ago

Hi @keegoid-nr thanks for the details on this issue.

How often does this issue come up for you and what's its impact to your application/service?

Have you identified a workaround in the meantime?

keegoid-nr commented 2 years ago

@kford-newrelic The issue happens 100% of the time.

The occurrence of escape characters in the NR_LAMBDA_MONITORING payload does not appear to cause any issues with New Relic's ingest of telemetry. It's possible we are doing some magic to parse the encoded telemetry and trim the escape chars automatically, because telemetry isn't missing from the New Relic UI.
When we try to manually decode the payload copied from CloudWatch logs, we have to account for escape characters in the payload.
The NR_LAMBDA_MONITORING payload ought to be standardized across agents. The NodeJS and Python agents do not add escape chars into the payload when written to the console.

NodeJS example payload:

[1,"NR_LAMBDA_MONITORING","H4sIAKGxdWIAA+1YWY/bNhB+968whD7akqhbfnOzW3SBoAlit31YLARKGrlsaEolKXudxf73UpK1kVXvAR9FGlcv1JDDOT7ODI+HwXCoLUHiFEusTYYPilY9mDNFVM0Er8WE4mWc4kkpxmsQcmxNbj9dX03fza+v7iZZyRJJcjb5vCwpxQw2Y0HFWFaMLE/hT4Gce23UyG2ZoxVwodpKyQ/vp/Pr2bxlgXtIypoH2IrwnC2ByYpv+vssel8bErVy9SfBBc9lnuS0Ixh52zG8UBK6GgMdmbqpqdHHikXb9V2hwUkSbTtv687hkCnnRtt/5LkoCEwLuXqA0L7eMGh72/ndv+FWVftpDC+hMmxWFkXOJY4JJXJj/FK7afzWWG4gR+tMexx1iNsdeWi0Q5rHkB3q7un/bnS8V9OioCTB1UobHwpghC2Mq5LXHQf6qSPb8gPPR+EZB0zke7ZlI8t3FIcTev8KQjeMSIIp+dKQxwJlOf7JukwPmWZ4ehSmH28MAbJJ+Z8xSynw7ykB5hwzgetyKIwZLKo6Jw5z0DqCdE7v4KzA7Hql3DHmucS0/hUzgEPD9dtavme8U5vUf9K7D/IP4J1YNH7a7tHGixv6gYXHNE3Lsh3XO9uAo4ehbbmBHdoejM3gDBDV6z4nS/gfq1exMjClFw3AU7BcFArtAeXHzTsVAMCNX9lnlq/Zs+2lhckhANWhdVEo9Y+EDNONJIloNt3v8njxoo/f0CFj0G23HmtCnY0iqIx96f78FQiNgwC+ygmPBPlSwWGp4PqqX6tliUhUq6uu9IOe42+6XctN0QCtjNNGvTGOE7hJq2HLgSyG2PJt27JC5KpbDUozK/N9bPqxifpTFyWp52HTCwLXi103DMGzYY+Kdi9oFPleEiYhznDqZbYVxv0JAi8LChWr5CX0BgtOcq7ipEJD91UOuqP9YXXgua4nTV0/YZHzSp22AAacJP/wT+1tQiqb6zeX9g0kCJ0eX9reWSfPlxCNcV0tON98zEn96rMHgQ6e+pkOsQ+71K5+vBZ68yCmn+mZbEcVh79KxdSEDmSugzInHXux540dL4jHQRiG49gJkiTwUxVW/j4hW3uTnKYzifkzyDIewX1CS0FWELXLFS0JpUTUq9YsWRe1V+oB3haz42qCysSTF4U1xFe7AVmlvR1Yb4zuPt9p0y59k2lPJ8uXMqqtfh2b+tqA85wrngxTsSfbji2Qr9W7V0vabkG89KR9S/oNqrHHwd9DSA6N3hcAAA=="]

This is how they generate it: https://github.com/newrelic/node-newrelic/blob/a07d045bdce148f60c0842ee4859052bbbdeff8a/lib/collector/serverless.js#L299-L315

workato-integration[bot] commented 1 year ago

https://issues.newrelic.com/browse/NEWRELIC-4082

kford-newrelic commented 1 year ago

Not selected for the Jan-Mar quarter.

newrelic / newrelic-lambda-tracer-java