newrelic / newrelic-java-agent

The New Relic Java agent
Apache License 2.0
199 stars 143 forks source link

Add support for grpc-netty #2044

Open jasonjkeller opened 3 weeks ago

jasonjkeller commented 3 weeks ago

To support HTTP/2, the netty-4.1.16 instrumentation wraps the Http2Headers to get info about the request and response. An issue (https://github.com/newrelic/newrelic-java-agent/issues/2024) was filed due to Http2Headers methods throwing UnsupportedOperationExceptions when the netty instrumentation would try to retrieve the required info.

Upon further examination, it was found that grpc-netty provides an io.grpc.netty.AbstractHttp2Headers implementation of Http2Headers that throws an UnsupportedOperationException from most methods. The grpc-netty library does have an io.grpc.netty.GrpcHttp2HeadersUtils class which provides concrete implementations of AbstractHttp2Headers:

abstract static class GrpcHttp2InboundHeaders extends AbstractHttp2Headers

static final class GrpcHttp2RequestHeaders extends GrpcHttp2InboundHeaders

static final class GrpcHttp2ResponseHeaders extends GrpcHttp2InboundHeaders

The issue with UnsupportedOperationExceptions has been resolved in https://github.com/newrelic/newrelic-java-agent/pull/2042 but we still need to add proper instrumentation for grpc-netty to be able to get the request and response info from the GrpcHttp2RequestHeaders and GrpcHttp2ResponseHeaders, respectively.


Relevant info from the customer:

The request.uri is always unknown. I am using jetcd-core:0.7.5

Screenshot 2024-09-06 at 12 10 31 PM
2024-09-06T21:05:37,784+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers path: null
at io.grpc.netty.AbstractHttp2Headers.path(AbstractHttp2Headers.java:540) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers authority: null
at io.grpc.netty.AbstractHttp2Headers.authority(AbstractHttp2Headers.java:530) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,791+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,856+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers content-length: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,857+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[?:?]
2024-09-06T21:05:37,859+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers content-type: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[?:?]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[?:?]
2024-09-06T21:05:37,860+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]

Repro instructions from https://github.com/newrelic/newrelic-java-agent/issues/2024#issuecomment-2335179636

Steps

  1. Run etcd. (docker compose up -d)
  2. create jar (mvn clean package)
  3. run the jar with newrelic agent .(i have used the jar mentioned in this issue which has try catch added https://github.com/newrelic/newrelic-java-agent/actions/runs/10725431981) java -javaagent:replaceme -Dnewrelic.config.app_name=replaceme -Dnewrelic.config.log_level=finest -Dnewrelic.config.log_file_name=STDOUT -Dnewrelic.config.license_key=replaceme -jar ./target/JetcdNewrelic-1.0-SNAPSHOT-jar-with-dependencies.jar
  4. check the console logs for error

JetcdNewrelic.zip

workato-integration[bot] commented 3 weeks ago

https://new-relic.atlassian.net/browse/NR-310733

kford-newrelic commented 3 days ago

Net new instrumentation