search

opentracing-contrib / java-specialagent

Automatic instrumentation for 3rd-party libraries in Java applications with OpenTracing.
Apache License 2.0
185 stars 46 forks source link

unprotected WeakHashMap in servlet rule leads to 100% CPU intermittently #578

Closed randallt closed 4 years ago

randallt commented 4 years ago

We have had at least two services hit this issue where CPU usage quickly jumps up to 100%, essentially leading to an outage for that host. Thread dumps show service threads stuck on

java.util.WeakHashMap.put(WeakHashMap.java:453)
at io.opentracing.contrib.specialagent.rule.servlet.FilterAgentIntercept.setStatusCode(FilterAgentIntercept.java:95) at org.springframework.security.web.util.OnCommittedResponseWrapper.sendRedirect(OnCommittedResponseWrapper.java:128) at org.springframework.security.web.DefaultRedirectStrategy.sendRedirect(DefaultRedirectStrategy.java:57) .. Sometimes the callers of setStatusCode are different, but they seem to usually be about redirects (might just be this workload).

Use of an unprotected Weak/HashMap can easily cause this. I'm surprised we haven't caught this before now.

This needs to be fixed by either synchronizing the WeakHashMap access or by using a ConcurrentHashMap with WeakReferences somehow.

randallt commented 4 years ago

I have changed my fork to use

private static final Map<ServletResponse,Integer> servletResponseToStatus = Collections.synchronizedMap(new WeakHashMap<>());

for now.