Open mihalyr opened 1 week ago
Note that while the above log says it is a warning, this breaks the server completely, so it's kind of a major problem.
Some good news, it seems this does not happen when I call Jersey paths, but if I call an path that is not found on the server, I receive a NPE from the API instead of the 404.
It would be still nice to fix this so it doesn't crash and returns 404 as expected. Metric instrumentation should not leak errors up to the servlet response, IMO.
Odd, as 404 case is handled right before this NPE occurs: https://github.com/eclipse-ee4j/jersey/blob/ed233c5af83901758d1ddc4e9555575c4c66986b/ext/micrometer/src/main/java/org/glassfish/jersey/micrometer/server/JerseyKeyValues.java#L91
So for 404, it should never get to matchingPattern
. Do you know why the URI_NOT_FOUND
is not returned?
So for 404, it should never get to matchingPattern. Do you know why the URI_NOT_FOUND is not returned?
The response
is null
in my case, so it never gets to check for 404.
This is how the RequestEvent looks like:
This is how my API response looks like when I remove jersey-micrometer
from the build:
✗ curl -v http://localhost:8080/
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Tue, 02 Jul 2024 12:02:32 GMT
< Vary: Accept-Encoding
< Vary: Origin
< Content-Length: 0
<
* Connection #0 to host localhost left intact
But when I use jersey-micrometer
I get an HTTP 500 and a runtime exception instead bubbling up from the metrics library into my API:
✗ curl -v http://localhost:8080/
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 500 Server Error
< Date: Tue, 02 Jul 2024 12:04:51 GMT
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 6603
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 jakarta.servlet.ServletException: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "matchingPattern" is null</title>
</head>
<body><h2>HTTP ERROR 500 jakarta.servlet.ServletException: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "matchingPattern" is null</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>jakarta.servlet.ServletException: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "matchingPattern" is null</td></tr>
<tr><th>SERVLET:</th><td>com.app.server.api.MyJerseyConfig</td></tr>
<tr><th>CAUSED BY:</th><td>java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "matchingPattern" is null</td></tr>
</table>
<h3>Caused by:</h3><pre>java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "matchingPattern" is null
at org.glassfish.jersey.micrometer.server.JerseyKeyValues.uri(JerseyKeyValues.java:96)
at org.glassfish.jersey.micrometer.server.DefaultJerseyObservationConvention.getLowCardinalityKeyValues(DefaultJerseyObservationConvention.java:43)
at org.glassfish.jersey.micrometer.server.DefaultJerseyObservationConvention.getLowCardinalityKeyValues(DefaultJerseyObservationConvention.java:30)
at io.micrometer.observation.SimpleObservation.start(SimpleObservation.java:152)
at io.micrometer.observation.docs.ObservationDocumentation.start(ObservationDocumentation.java:241)
at org.glassfish.jersey.micrometer.server.ObservationRequestEventListener.startObservation(ObservationRequestEventListener.java:99)
at org.glassfish.jersey.micrometer.server.ObservationRequestEventListener.onEvent(ObservationRequestEventListener.java:71)
at org.glassfish.jersey.server.internal.monitoring.CompositeRequestEventListener.onEvent(CompositeRequestEventListener.java:47)
at org.glassfish.jersey.server.internal.process.RequestProcessingContext.triggerEvent(RequestProcessingContext.java:203)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:442)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:282)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)
at org.eclipse.jetty.ee10.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:195)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:317)
at org.eclipse.jetty.ee10.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:270)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:703)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
at org.eclipse.jetty.server.Server.handle(Server.java:181)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:403)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
</pre>
</body>
</html>
* Connection #0 to host localhost left intact
I believe, there are two problems:
And possibly a third problem is that the 404 check doesn't cover the case when the 404 is from a WebApplicationException like with Jersey:
Here is the stacktrace of that 404 exception in case it helps:
jakarta.ws.rs.NotFoundException: HTTP 404 Not Found
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)
at org.eclipse.jetty.ee10.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:195)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:317)
at org.eclipse.jetty.ee10.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:270)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:703)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
at org.eclipse.jetty.server.Server.handle(Server.java:181)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:403)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
Forgot to add the library versions I am using:
After upgrading to Spring Boot 3.3 with Micrometer 1.13 I followed the upgrade instructions to add a dependency on
org.glassfish.jersey.ext:jersey-micrometer
I noticed that after the upgrade, this new library is throwing NPEs from the servlet:
It seems the library does not handle null values properly and causing unexpected failures during servlet processing.
Here is the problematic part: https://github.com/eclipse-ee4j/jersey/blob/ed233c5af83901758d1ddc4e9555575c4c66986b/ext/micrometer/src/main/java/org/glassfish/jersey/micrometer/server/JerseyKeyValues.java#L95-L98
Where
JerseyTags.getMatchingPattern
is a nullable method, but the return value is not checked and causes NPE at runtime.