To support HTTP/2, the netty-4.1.16 instrumentation wraps the Http2Headers to get info about the request and response. An issue (https://github.com/newrelic/newrelic-java-agent/issues/2024) was filed due to Http2Headers methods throwing UnsupportedOperationExceptions when the netty instrumentation would try to retrieve the required info.
Upon further examination, it was found that grpc-netty provides an io.grpc.netty.AbstractHttp2Headers implementation of Http2Headers that throws an UnsupportedOperationException from most methods. The grpc-netty library does have an io.grpc.netty.GrpcHttp2HeadersUtils class which provides concrete implementations of AbstractHttp2Headers:
abstract static class GrpcHttp2InboundHeaders extends AbstractHttp2Headers
static final class GrpcHttp2RequestHeaders extends GrpcHttp2InboundHeaders
static final class GrpcHttp2ResponseHeaders extends GrpcHttp2InboundHeaders
The issue with UnsupportedOperationExceptions has been resolved in https://github.com/newrelic/newrelic-java-agent/pull/2042 but we still need to add proper instrumentation for grpc-netty to be able to get the request and response info from the GrpcHttp2RequestHeaders and GrpcHttp2ResponseHeaders, respectively.
Relevant info from the customer:
The request.uri is always unknown. I am using jetcd-core:0.7.5
2024-09-06T21:05:37,784+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers path: null
at io.grpc.netty.AbstractHttp2Headers.path(AbstractHttp2Headers.java:540) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers authority: null
at io.grpc.netty.AbstractHttp2Headers.authority(AbstractHttp2Headers.java:530) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,791+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,856+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers content-length: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,857+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[?:?]
2024-09-06T21:05:37,859+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers content-type: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[?:?]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[?:?]
2024-09-06T21:05:37,860+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]
run the jar with newrelic agent .(i have used the jar mentioned in this issue which has try catch added https://github.com/newrelic/newrelic-java-agent/actions/runs/10725431981) java -javaagent:replaceme -Dnewrelic.config.app_name=replaceme -Dnewrelic.config.log_level=finest -Dnewrelic.config.log_file_name=STDOUT -Dnewrelic.config.license_key=replaceme -jar ./target/JetcdNewrelic-1.0-SNAPSHOT-jar-with-dependencies.jar
To support HTTP/2, the
netty-4.1.16
instrumentation wraps theHttp2Headers
to get info about the request and response. An issue (https://github.com/newrelic/newrelic-java-agent/issues/2024) was filed due toHttp2Headers
methods throwingUnsupportedOperationException
s when the netty instrumentation would try to retrieve the required info.Upon further examination, it was found that
grpc-netty
provides anio.grpc.netty.AbstractHttp2Headers
implementation ofHttp2Headers
that throws anUnsupportedOperationException
from most methods. Thegrpc-netty
library does have anio.grpc.netty.GrpcHttp2HeadersUtils
class which provides concrete implementations ofAbstractHttp2Headers
:The issue with
UnsupportedOperationException
s has been resolved in https://github.com/newrelic/newrelic-java-agent/pull/2042 but we still need to add proper instrumentation forgrpc-netty
to be able to get the request and response info from theGrpcHttp2RequestHeaders
andGrpcHttp2ResponseHeaders
, respectively.Relevant info from the customer:
The
request.uri
is always unknown. I am usingjetcd-core:0.7.5
Repro instructions from https://github.com/newrelic/newrelic-java-agent/issues/2024#issuecomment-2335179636
Steps
docker compose up -d
)mvn clean package
)java -javaagent:replaceme -Dnewrelic.config.app_name=replaceme -Dnewrelic.config.log_level=finest -Dnewrelic.config.log_file_name=STDOUT -Dnewrelic.config.license_key=replaceme -jar ./target/JetcdNewrelic-1.0-SNAPSHOT-jar-with-dependencies.jar
JetcdNewrelic.zip