search

hapifhir / hapi-fhir

🔥 HAPI FHIR - Java API for HL7 FHIR Clients and Servers
http://hapifhir.io
Apache License 2.0
2k stars 1.31k forks source link

OutOfMemoryError: Java heap space - in ResponseHighlighterInterceptor on large responses #1872

Open rbhman opened 4 years ago

rbhman commented 4 years ago

When trying to view very large search responses in the browser (e.g. 188,770 bundle entries as per the log entry below) with ResponseHighlighterInterceptor enabled, I typically encounter an OutOfMemoryError at a random append statement:

... 14:23:54.065 INFO c.u.p.c.p.ObservationResourceProvider getAllObservations returning 188700 bundle entries 14:24:28.019 ERROR c.u.f.r.s.i.ExceptionHandlingInterceptor Failure during REST processing ca.uhn.fhir.rest.server.exceptions.InternalErrorException: Failure invoking interceptor for pointcut(s) SERVER_OUTGOING_RESPONSE at ca.uhn.fhir.interceptor.executor.InterceptorService$HookInvoker.invoke(InterceptorService.java:508) at ca.uhn.fhir.interceptor.executor.InterceptorService.doCallHooks(InterceptorService.java:271) at ca.uhn.fhir.interceptor.executor.InterceptorService.callHooks(InterceptorService.java:260) at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.callOutgoingResponseHook(BaseResourceReturningMethodBinding.java:443) at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.invokeServer(BaseResourceReturningMethodBinding.java:393) at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:991) at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:336) at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1675) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:445) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:459) at java.lang.StringBuilder.append(StringBuilder.java:166) at ca.uhn.fhir.rest.server.interceptor.ResponseHighlighterInterceptor.streamResponse(ResponseHighlighterInterceptor.java:656) at ca.uhn.fhir.rest.server.interceptor.ResponseHighlighterInterceptor.handleOutgoingResponse(ResponseHighlighterInterceptor.java:416) at ca.uhn.fhir.rest.server.interceptor.ResponseHighlighterInterceptor.outgoingResponse(ResponseHighlighterInterceptor.java:340) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at ca.uhn.fhir.interceptor.executor.InterceptorService$HookInvoker.invoke(InterceptorService.java:497) at ca.uhn.fhir.interceptor.executor.InterceptorService.doCallHooks(InterceptorService.java:271) at ca.uhn.fhir.interceptor.executor.InterceptorService.callHooks(InterceptorService.java:260) at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.callOutgoingResponseHook(BaseResourceReturningMethodBinding.java:443) at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.invokeServer(BaseResourceReturningMethodBinding.java:393) at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:991) at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:336) at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1675) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)

As an experiment I tried registering my own local version of ResponseHighlighterInterceptor with:

StringBuilder outputBuffer = new StringBuilder(encoded.length()); instead of: StringBuilder outputBuffer = new StringBuilder(); where applicable but this did not help.

Perhaps this need to stream to disk and then cleanup resources instead? At least above a certain threshold? Or better initial capacity estimates for StringBuilder?

sajithdil commented 4 years ago

just ran into this issue myself, did you figure out a solution for this?

rbhman commented 4 years ago

No - I just stopped using the interceptor.

On Wed, Jun 24, 2020 at 11:48 PM sajithdil notifications@github.com wrote:

just ran into this issue myself, did you figure out a solution for this?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jamesagnew/hapi-fhir/issues/1872#issuecomment-649198108, or unsubscribe https://github.com/notifications/unsubscribe-auth/APXLJKZIL2XQYEW6QLOFNUTRYLCHBANCNFSM4NK6TB6Q .