Open dadebuch opened 2 years ago
I also experienced "OutOfMemoryError: Java heap space" recently again. In my case the memory of the container was limited using mem_limit (https://docs.docker.com/compose/compose-file/compose-file-v2/#cpu-and-other-resources). Raising the limit allowed me to transfer bigger files. I think the problem on the provider side is that the data requested from a DSC is read into a string before being returned to the client. On the receiver side the data is also read into a string when it is received. Therefore the more data you transfer in one request the more memory you will need.
The OutOfMemorError happens on the consumer side. This is the log:
consumer_connector_1 2021-10-20 15:05:17,703 scheduling-1 INFO io.dataspaceconnector.service.usagecontrol.ScheduledDataRemoval - Scanning agreements...
2021-10-20 17:05:09
consumer_connector_1 java.lang.OutOfMemoryError: Java heap space
2021-10-20 17:05:09
consumer_connector_1 2021-10-20 15:05:09,165 https-jsse-nio-8080-exec-1 ERROR org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
2021-10-20 17:05:09
consumer_connector_1 at java.lang.Thread.run(Thread.java:829) [?:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189) [opentracing-web-servlet-filter-0.4.0.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.5.4.jar!/:2.5.4]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:178) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) [spring-security-web-5.5.2.jar!/:5.5.2]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.52.jar!/:?]
2021-10-20 17:05:09
consumer_connector_1 at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) [tomcat-embed-core-9.0.52.jar!/:4.0.1]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) [tomcat-embed-core-9.0.52.jar!/:4.0.1]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1135) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1324) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:141) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) [spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) ~[spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:167) ~[spring-web-5.3.9.jar!/:5.3.9]
2021-10-20 17:05:09
consumer_connector_1 java.lang.IllegalStateException: Could not resolve parameter [0] in public org.springframework.http.ResponseEntity<net.minidev.json.JSONObject> io.dataspaceconnector.controller.exceptionhandler.GlobalExceptionHandler.handleAnyException(java.lang.RuntimeException): No suitable resolver
2021-10-20 17:05:09
consumer_connector_1 2021-10-20 15:05:09,159 https-jsse-nio-8080-exec-1 WARN org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver - Failure in @ExceptionHandler io.dataspaceconnector.controller.exceptionhandler.GlobalExceptionHandler#handleAnyException(RuntimeException)
2021-10-20 17:05:05
consumer_connector_1 2021-10-20 15:05:05,985 https-jsse-nio-8080-exec-1 INFO de.fraunhofer.ids.messaging.protocol.http.IdsHttpService - Successfully received response to request.
2021-10-20 17:05:05
consumer_connector_1 2021-10-20 15:05:05,333 https-jsse-nio-8080-exec-1 INFO de.fraunhofer.ids.messaging.protocol.http.IdsHttpService - Sending request to https://ids.dwd-connector.mobilitydataspace.io/api/ids/data ...
2021-10-20 17:05:05
consumer_connector_1 2021-10-20 15:05:05,309 https-jsse-nio-8080-exec-1 INFO de.fraunhofer.ids.messaging.core.daps.aisec.AisecTokenManagerService - Successfully received DAT from DAPS.
2021-10-20 17:05:05
consumer_connector_1 2021-10-20 15:05:05,079 https-jsse-nio-8080-exec-1 WARN de.fraunhofer.ids.messaging.core.daps.TokenProviderService - Could not parse JWT! Treat JWT as having expired.
Remove RAM limitations: As described here and https://github.com/International-Data-Spaces-Association/DataspaceConnector/issues/181 the DSC caches received data in the RAM, until it is fully transferred. Transferable file size, thus depends on RAM availability on consumers side.
The discussion link at the top is to this discussion (archived in the old repo): #102
Discussed in https://github.com/International-Data-Spaces-Association/DataspaceConnector/discussions/677