As a reference, the Python Agent team has a write up on what they needed to do to address this problem within the Python agent.
This topic came up as part of customer escalation, who was having some challenges with New Relic when using AWS API Gateway configured in HTTP mode which executes a lambda. For this customer, there are actually a chain of API Gateway (HTTP) → Lambda → API Gateway (HTTP) → Lambda calls. When manually testing this use case to understand the behavior within New Relic, it was found that each lambda had their own trace_id’s even though the expected HTTP headers were being forwarded to the second lambda.
Acceptance Criteria
If both entities have the same trace_id and the service map for the distributed trace shows both lambdas, you are done and no changes are needed.
If both entities have different trace_id’s and the service map for the distributed trace only shows the lambda that you are viewing from, then there is an issue with propagating the trace_id between the lambdas that needs to be debugged and fixed.
Design Consideration/Limitations
N/A
Dependencies
In order to run the test, it does take a bit of setup within AWS.
Set up the “backend” API Gateway and Lambda
Create a lambda in Ruby that will return the contents of the incoming event as JSON as part of the response (backend lambda)
Create an API Gateway in HTTP mode (not REST) that will forward the request to the lambda you just created
Test for basic functionality
Update the lambda to use the appropriate New Relic lambda layer
Test and ensure the lambda entity is defined in New Relic as well as the distributed trace being available
Set up the “frontend” API Gateway and Lambda
Create a lambda in Ruby that will make an HTTP request to the “backend” API Gateway URL. The response should include the event for the “frontend” lambda as well as the results reported by the “backend” call
Create an API Gateway in HTTP mode (not REST) that will forward the request to the lambda you just created
Test for basic functionality
Update the lambda to use the appropriate New Relic lambda layer
Test and ensure the lambda entity is defined in New Relic as well as the distributed trace being available
With the above defined and working, run curl commands to generate several test distributed traces. For the same external request, are both entities using the same trace_id or are they generating different trace_ids?
workato-integration[bot]
commented
4 weeks ago
Description
As a reference, the Python Agent team has a write up on what they needed to do to address this problem within the Python agent.
This topic came up as part of customer escalation, who was having some challenges with New Relic when using AWS API Gateway configured in HTTP mode which executes a lambda. For this customer, there are actually a chain of API Gateway (HTTP) → Lambda → API Gateway (HTTP) → Lambda calls. When manually testing this use case to understand the behavior within New Relic, it was found that each lambda had their own trace_id’s even though the expected HTTP headers were being forwarded to the second lambda.
Acceptance Criteria
If both entities have the same trace_id and the service map for the distributed trace shows both lambdas, you are done and no changes are needed.
If both entities have different trace_id’s and the service map for the distributed trace only shows the lambda that you are viewing from, then there is an issue with propagating the trace_id between the lambdas that needs to be debugged and fixed.
Design Consideration/Limitations
N/A
Dependencies
In order to run the test, it does take a bit of setup within AWS.
Set up the “backend” API Gateway and Lambda
Set up the “frontend” API Gateway and Lambda
With the above defined and working, run curl commands to generate several test distributed traces. For the same external request, are both entities using the same trace_id or are they generating different trace_ids?
Additional context
See NR-265378 for Python example code and output