Issue with grpc 1.4.x agent not properly generating and propagating trace information

[mbazos]

When upgrading gRPC from 1.3.x to 1.4.x the handling of trace identifiers no longer works. Meaning the trace information is always empty for gRPC 1.4.x but works fine for 1.3.x

Regardless of the version of grpc the tracing should work

Steps to Reproduce

Attached is a spring-boot application that has a grpc server and client. You can run the server setting the appropriate new relic agent settings and then run com.priceline.grpc.demo.demo.ClientGrpcRunner and you will see output similar to:

Hello Mike Bazos your trace id is[]

Your Environment

Azul Java 17 (Please see the attached logs which include the same program running with grpc 1.3.x and 1.4.x) grpc_1_4_logs.txt grpc_1_3_logs.txt grpc_new_relic_issue.zip

[workato-integration[bot]]

https://issues.newrelic.com/browse/NEWRELIC-4191

[kford-newrelic]

@mbazos Thank you for the detailed info (and repro) on this issue. Do you currently have a workaround to this issue that you're using?

[atyulmankov]

@mbazos Thank you for the detailed info (and repro) on this issue. Do you currently have a workaround to this issue that you're using?

@kford-newrelic I am not aware of any workarounds, and I am waiting for resolution on this as well.

[mbazos]

Yeah @kford-newrelic I am not aware of any workarounds, I didn't have time to dig deeper into the new relic agent code to see exactly what the problem is.

[kford-newrelic]

Understood, thank you for the feedback. We don't have the capacity to take this up in the current Oct-Dec quarter but we'll consider this for our next quarter's planning.

[mbazos]

You can get this from the logs but just wanted to let people know the version of grpc that the issue is caused by based on my logs is grpc 1.44.1. I see there is instrumentation for 1.4.0 & 1.40.0 https://github.com/newrelic/newrelic-java-agent/tree/main/instrumentation/grpc-1.4.0

[mbazos]

So I think I found the root problem but just not sure how New Relic wants to go about fixing this. I do see one difference this appears to be the problem:

New Relic Instrumentation class io.grpc.internal.ServerCallImpl_Instrumentation

@Weave(originalName = "io.grpc.internal.ServerCallImpl")
final class ServerCallImpl_Instrumentation {

    ServerStreamListener newServerStreamListener(ServerCallListener_Instrumentation listener) {
        // This is the point where a request comes into grpc
        // Store a token on the listener so we can bring the transaction into customer code
        listener.token = AgentBridge.getAgent().getTransaction().getToken();
        return Weaver.callOriginal();
    }
}

1.44.1:

    ServerStreamListener newServerStreamListener(ServerCall.Listener<ReqT> listener) {
        return new ServerStreamListenerImpl(this, listener, this.context);
    }

Basically the input parameters changed. So I assume the instrumentation will never get called

[kford-newrelic]

@mbazos Great analysis - thank you!

Discussing with the team to see if we can build on your findings

[workato-integration[bot]]

Work has been completed on this issue.