newtork
commented
3 months ago Hallo Stephan,
I'm in discussion with the team as to what could be the cause for the observed memory leak. As far as I can tell we didn't change anything significant from 5.4.0 to 5.5.0 with relation to Destination and HttpClient usage.
However I could imagine that the CSRF Token connections not being closed eagerly to be one of the culprits. I drafted a PR to consume the HttpEntity and therefore hopefully close the connection upon evaluation. Of course this doesn't help you now.
However, in case you don't need CSRF Token interaction since it's failing for you anyway(?), maybe you could try to disable it altogether?
var request =
new CreateRequestBuilder<>(
COMMAND_ENDPOINT, createEntity(payload), BDTS_COMMAND_ENTITY_NAME)
+ .withoutCsrfToken()
.toRequest();If not permanently for all tenants, maybe you could still confirm, this reduces the memory consumption and solves the leak?
Kind regards Alexnder
stephangutknecht
KavithaSiva
maksimyugai
MatKuhr
CharlesDuboisSAP
Issue Description
In btc-cross-gateway-abap-srv, we are using cloud-sdk to call on-premise systems via BTP destination. With release 5.5.0, we realized calling on-premise systems seems to steadily increase the used memory of our app instance. The pattern we see is that first, CSRF token requests are send (as token is invalid) and afterwards, the actual request is send to the destination. The memory on our app instance is not decreasing after processing is done.
We have seen exceptions like this:
Time out occurred because of a probable connection leak. Please execute your request with try-with-resources to ensure resources are properly closed.If you are using the OData client instead to execute your request, explicitly consume the entity of the associated HttpResponse using EntityUtils.consume(httpEntity)With release 5.4.0, we do not see these issues. Still, memory seems to be decreasing rather slowly, we will further investigate as well on our side.
Impact / Priority
Affected development phase: Release
Impact: Blocked
Timeline: Go-Live is in 3 weeks
Error Message
We do see client failures when services is out-of-memory. It is however not crashing.
Maven Dependency
com.sap.btc.cross.gateway.abap:btc-cross-gateway-abap-srv:jar:1.0.0 +- com.sap.cds:cds-starter-spring-boot:jar:2.6.1:compile | +- com.sap.cds:cds-services-api:jar:2.6.1:compile | | +- com.sap.cds:cds4j-api:jar:2.6.1:compile | | - com.sap.cloud.environment.servicebinding.api:java-core-api:jar:0.10.3:compile | +- com.sap.cds:cds-framework-spring-boot:jar:2.6.1:runtime | +- org.springframework.boot:spring-boot-starter-web:jar:3.2.2:compile | | +- org.springframework.boot:spring-boot-starter-json:jar:3.2.2:compile | | | +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.3:compile | | | - com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.15.3:compile | | +- org.springframework.boot:spring-boot-starter-tomcat:jar:3.2.2:compile | | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:10.1.18:compile | | | +- org.apache.tomcat.embed:tomcat-embed-el:jar:10.1.18:compile | | | - org.apache.tomcat.embed:tomcat-embed-websocket:jar:10.1.18:compile | | +- org.springframework:spring-web:jar:6.1.3:compile | | - org.springframework:spring-webmvc:jar:6.1.3:compile | - org.yaml:snakeyaml:jar:2.2:compile +- com.sap.cds:cds-adapter-odata-v4:jar:2.6.1:runtime | +- com.sap.cds:cds-adapter-api:jar:2.6.1:compile | +- com.sap.cds:cds-services-utils:jar:2.6.1:compile | | +- io.opentelemetry:opentelemetry-api:jar:1.31.0:compile | | | - io.opentelemetry:opentelemetry-context:jar:1.31.0:compile | | +- com.sap.cloud.security.xsuaa:token-client:jar:3.3.4:compile | | - com.sap.cloud.mt:tools:jar:2.6.1:compile | +- com.sap.cds:cds4j-core:jar:2.6.1:compile | | - com.github.ben-manes.caffeine:caffeine:jar:3.1.8:compile | +- com.sap.cloud.mt:cds-mtx:jar:2.6.1:compile | | - commons-io:commons-io:jar:2.11.0:compile | - com.sap.cds.repackaged:odata-v4-lib:jar:2.6.1:runtime | +- commons-codec:commons-codec:jar:1.16.0:compile | +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.15.3:runtime | | - org.codehaus.woodstox:stax2-api:jar:4.2.1:runtime | - com.fasterxml:aalto-xml:jar:1.3.2:runtime +- com.sap.cloud.tenantlifecycle:euporie-dwc-integration:jar:2.0.0-20240221090523_b27f38d3c7984f90a66471f22b4d5c106ea0c260:compile | +- io.pivotal.cfenv:java-cfenv-boot:jar:3.1.5:compile | | +- io.pivotal.cfenv:java-cfenv-jdbc:jar:3.1.5:compile | | - org.springframework.boot:spring-boot:jar:3.2.2:compile | +- org.json:json:jar:20231013:compile | +- org.apache.commons:commons-lang3:jar:3.14.0:compile | +- com.sap.cloud.environment.servicebinding:java-sap-vcap-services:jar:0.10.3:compile | +- com.sap.cloud.environment.servicebinding:java-sap-service-operator:jar:0.10.3:compile | +- com.sap.cloud.tenantlifecycle:euporie-api:jar:2.0.0-20240223143312_5cb76ce618f22654cb288b49024f1843032b854c:compile | | - com.sap.cloud.tenantlifecycle:euporie-api-domain:jar:2.0.0-20240223143312_5cb76ce618f22654cb288b49024f1843032b854c:compile | +- com.sap.cloud.tenantlifecycle:hegemone-specs:jar:2.0.0-20240223131614_684d4d07779bd192d6b5d79ea703e93830438085:compile | +- com.sap.cloud.tenantlifecycle:euporie-authentication:jar:2.0.0-20240214115019_af60650aad511598c8e5aac76f412d341ead5275:compile | | - org.reflections:reflections:jar:0.10.2:compile | | - org.javassist:javassist:jar:3.25.0-GA:compile | - com.sap.dwc:util-model:jar:2.3.1:compile +- com.sap.cloud.sdk.datamodel:odata-core:jar:5.4.0:compile | +- com.sap.cloud.sdk.datamodel:odata-client:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:cloudplatform-core:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:cloudplatform-connectivity:jar:5.4.0:compile | | +- com.sap.cloud.sdk.cloudplatform:resilience-api:jar:5.4.0:compile | | +- com.mikesamuel:json-sanitizer:jar:1.2.3:compile | | - com.auth0:java-jwt:jar:4.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:connectivity-apache-httpclient4:jar:5.4.0:compile | | - com.sap.cloud.sdk.cloudplatform:caching:jar:5.4.0:compile | +- com.sap.cloud.sdk.datamodel:fluent-result:jar:5.4.0:compile | +- org.slf4j:jcl-over-slf4j:jar:2.0.11:runtime | +- com.google.guava:guava:jar:32.0.1-jre:compile | | +- com.google.guava:failureaccess:jar:1.0.1:compile | | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile | | +- org.checkerframework:checker-qual:jar:3.42.0:compile | | +- com.google.errorprone:error_prone_annotations:jar:2.14.0:compile | | - com.google.j2objc:j2objc-annotations:jar:2.8:compile | +- com.google.code.gson:gson:jar:2.10.1:compile | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.3:compile | +- com.fasterxml.jackson.core:jackson-core:jar:2.15.3:compile | +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.3:compile | +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile | +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile | - io.vavr:vavr:jar:0.10.4:compile | - io.vavr:vavr-match:jar:0.10.4:compile +- com.sap.cloud.sdk.datamodel:odata-v4-core:jar:5.4.0:compile +- com.sap.cds:cds-services-impl:jar:2.6.1:compile | +- com.sap.cds:cds-services-messaging:jar:2.6.1:compile | | - jakarta.jms:jakarta.jms-api:jar:3.1.0:compile | - com.sap.cloud.environment.servicebinding.api:java-access-api:jar:0.10.3:compile +- com.sap.cds:cds-feature-mt:jar:2.6.1:compile | +- com.sap.cloud.mt:multi-tenant-runtime:jar:2.6.1:compile | - com.sap.cloud.mt:multi-tenant-subscription:jar:2.6.1:compile | - com.sap.cloud.instancemanager:client:jar:3.14.0:compile +- com.sap.cds:cds-feature-cloudfoundry:jar:2.6.1:compile +- org.apache.commons:commons-csv:jar:1.10.0:compile +- com.sap.calm.x:uam-lib:jar:1.0.0-20240223094741_a7af5fa4a6cd94928ddc44f761cb36e5e2b47e14:compile | +- jakarta.servlet:jakarta.servlet-api:jar:6.0.0:compile | - com.sap.cloud.sdk.cloudplatform:resilience4j:jar:5.4.0:compile | +- io.github.resilience4j:resilience4j-circuitbreaker:jar:2.2.0:compile | | - io.github.resilience4j:resilience4j-core:jar:2.2.0:compile | +- io.github.resilience4j:resilience4j-bulkhead:jar:2.2.0:compile | +- io.github.resilience4j:resilience4j-timelimiter:jar:2.2.0:compile | +- io.github.resilience4j:resilience4j-retry:jar:2.2.0:compile | +- io.github.resilience4j:resilience4j-ratelimiter:jar:2.2.0:compile | - javax.cache:cache-api:jar:1.1.1:compile +- org.projectlombok:lombok:jar:1.18.30:provided +- com.sap.hcp.cf.logging:cf-java-logging-support-logback:jar:3.7.0:compile | - com.sap.hcp.cf.logging:cf-java-logging-support-core:jar:3.7.0:compile | - com.fasterxml.jackson.jr:jackson-jr-objects:jar:2.15.3:compile +- com.sap.cds:cds-integration-cloud-sdk:jar:2.6.1:compile | +- com.sap.cloud.sdk.cloudplatform:tenant:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:security:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:connectivity-oauth:jar:5.4.0:compile | | +- com.sap.cloud.environment.servicebinding.api:java-consumption-api:jar:0.10.3:compile | | +- com.sap.cloud.security:java-api:jar:3.3.4:compile | | - com.sap.cloud.security:java-security:jar:3.3.4:compile | - com.sap.cloud.sdk.frameworks:resilience4j:jar:4.29.0:compile +- org.springframework.security:spring-security-test:jar:6.2.1:test | +- org.springframework.security:spring-security-core:jar:6.2.1:compile | | +- org.springframework.security:spring-security-crypto:jar:6.2.1:compile | | +- org.springframework:spring-aop:jar:6.1.3:compile | | +- org.springframework:spring-beans:jar:6.1.3:compile | | +- org.springframework:spring-context:jar:6.1.3:compile | | - org.springframework:spring-expression:jar:6.1.3:compile | +- org.springframework.security:spring-security-web:jar:6.2.1:compile | +- org.springframework:spring-core:jar:6.1.3:compile | | - org.springframework:spring-jcl:jar:6.1.3:compile | - org.springframework:spring-test:jar:6.1.3:test +- com.sap.xs.auditlog:audit-java-client-api:jar:2.0.19:compile +- com.sap.xs.auditlog:audit-java-client-impl:jar:2.0.19:compile | +- com.sap.xs.auditlog:auditlog-common:jar:2.0.52:compile | - com.sap.xs.java:xs-user-holder:jar:1.8.3:compile | - com.sap.xs2.security:java-container-security-api:jar:0.33.18:compile | - com.sap.cloud.security.xssec:api:jar:1.0.2:compile +- com.sap.xs.java:xs-env:jar:1.8.5:compile +- org.slf4j:slf4j-api:jar:2.0.11:compile +- com.sap.dwc:util-cap:jar:2.3.1:compile | +- org.apache.httpcomponents.client5:httpclient5:jar:5.2.3:compile | | +- org.apache.httpcomponents.core5:httpcore5:jar:5.2.4:compile | | - org.apache.httpcomponents.core5:httpcore5-h2:jar:5.2.4:compile | - com.sap.dwc.commons:commons-util:jar:2.19.0:compile +- com.sap.dwc:util-headers:jar:2.3.1:compile +- com.sap.dwc:util-product-config:jar:2.3.1:compile +- com.sap.dwc:util-mutual-authentication:jar:2.3.1:compile | +- org.bouncycastle:bcprov-jdk18on:jar:1.74:compile | - org.bouncycastle:bcpkix-jdk18on:jar:1.75:compile | - org.bouncycastle:bcutil-jdk18on:jar:1.75:compile +- com.sap.dwc:util-routing:jar:2.3.1:compile +- com.sap.calm.x:dwc-foundation:jar:2.1.0-20240226114504_323cb0ee25bfc3fc737165be74d6e691ba02ecc0:compile | +- com.sap.cds:cds-starter-spring-boot-odata:jar:2.6.1:compile | +- jakarta.annotation:jakarta.annotation-api:jar:2.1.1:compile | +- jakarta.management.j2ee:jakarta.management.j2ee-api:jar:1.1.4:compile | +- com.googlecode.owasp-java-html-sanitizer:owasp-java-html-sanitizer:jar:20220608.1:compile | +- com.sap.cloud.sdk:sdk-core:jar:5.4.0:compile | | +- com.sap.cloud.sdk.cloudplatform:connectivity-destination-service:jar:5.4.0:compile | | - com.sap.cloud.sdk.cloudplatform:servlet-jakarta:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:connectivity-dwc:jar:5.4.0:compile | +- com.sap.cloud.sdk.cloudplatform:sap-passport:jar:4.28.0:compile | | - com.sap.core.jdsr:com.sap.js.passport.api:jar:1.8.0:compile | +- com.sap.cloud.sdk.cloudplatform:auditlog-scp-cf:jar:4.29.0:compile | | - com.sap.cloud.sdk.cloudplatform:auditlog:jar:4.29.0:compile | +- com.sap.cp.auditlog:audit-java-client-impl:jar:2.6.0:compile | | +- com.sap.cloud.security:env:jar:3.3.4:compile | | - com.sap.cp.auditlog:auditlog-common:jar:2.2.16:compile | +- com.sap.cp.auditlog:audit-java-client-api:jar:2.6.0:compile | +- org.springframework.boot:spring-boot-starter-security:jar:3.2.2:compile | | - org.springframework.security:spring-security-config:jar:6.2.1:compile | - org.jsoup:jsoup:jar:1.17.2:compile +- org.springframework.boot:spring-boot-starter-actuator:jar:3.2.2:compile | +- org.springframework.boot:spring-boot-starter:jar:3.2.2:compile | | +- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.2:compile | | - org.springframework.boot:spring-boot-starter-logging:jar:3.2.2:compile | | +- ch.qos.logback:logback-classic:jar:1.4.14:compile | | | - ch.qos.logback:logback-core:jar:1.4.14:compile | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.21.1:compile | | | - org.apache.logging.log4j:log4j-api:jar:2.21.1:compile | | - org.slf4j:jul-to-slf4j:jar:2.0.11:compile | +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:3.2.2:compile | | - org.springframework.boot:spring-boot-actuator:jar:3.2.2:compile | +- io.micrometer:micrometer-observation:jar:1.12.2:compile | | - io.micrometer:micrometer-commons:jar:1.12.2:compile | - io.micrometer:micrometer-jakarta9:jar:1.12.2:compile | - io.micrometer:micrometer-core:jar:1.12.2:compile | +- org.hdrhistogram:HdrHistogram:jar:2.1.12:runtime | - org.latencyutils:LatencyUtils:jar:2.0.3:runtime +- com.sap.cumulus.jiralinking:jira-annotation:jar:3.5.3:test +- com.sap.calm.metering:calm-metering-lib:jar:2.0.0-20240222062514_8b51388995c7417ff0ea946e10efa94717f572d0:compile | +- org.springframework.boot:spring-boot-starter-webflux:jar:3.2.2:compile | | +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:3.2.2:compile | | | - io.projectreactor.netty:reactor-netty-http:jar:1.1.15:compile | | | +- io.netty:netty-codec-http:jar:4.1.105.Final:compile | | | | +- io.netty:netty-common:jar:4.1.105.Final:compile | | | | +- io.netty:netty-buffer:jar:4.1.105.Final:compile | | | | +- io.netty:netty-transport:jar:4.1.105.Final:compile | | | | +- io.netty:netty-codec:jar:4.1.105.Final:compile | | | | - io.netty:netty-handler:jar:4.1.105.Final:compile | | | +- io.netty:netty-codec-http2:jar:4.1.105.Final:compile | | | +- io.netty:netty-resolver-dns:jar:4.1.105.Final:compile | | | | +- io.netty:netty-resolver:jar:4.1.105.Final:compile | | | | - io.netty:netty-codec-dns:jar:4.1.105.Final:compile | | | +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.105.Final:compile | | | | - io.netty:netty-resolver-dns-classes-macos:jar:4.1.105.Final:compile | | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.105.Final:compile | | | | +- io.netty:netty-transport-native-unix-common:jar:4.1.105.Final:compile | | | | - io.netty:netty-transport-classes-epoll:jar:4.1.105.Final:compile | | | - io.projectreactor.netty:reactor-netty-core:jar:1.1.15:compile | | | - io.netty:netty-handler-proxy:jar:4.1.105.Final:compile | | | - io.netty:netty-codec-socks:jar:4.1.105.Final:compile | | - org.springframework:spring-webflux:jar:6.1.3:compile | | - io.projectreactor:reactor-core:jar:3.6.2:compile | | - org.reactivestreams:reactive-streams:jar:1.0.4:compile | +- org.springframework.boot:spring-boot-starter-aop:jar:3.2.2:compile | | - org.aspectj:aspectjweaver:jar:1.9.21:compile | - com.sap.cloud.sdk.cloudplatform:resilience:jar:5.4.0:compile +- com.sap.calm.x:calm-kafka-clientlib:jar:2.1.0-20240220141031_e1d18dc1b3a447c6c3f87bd95dcb3008600749fe:compile | +- io.cloudevents:cloudevents-json-jackson:jar:2.5.0:compile | | - io.cloudevents:cloudevents-core:jar:2.5.0:compile | | - io.cloudevents:cloudevents-api:jar:2.5.0:compile | +- io.cloudevents:cloudevents-kafka:jar:2.5.0:compile | | - org.apache.kafka:kafka-clients:jar:3.6.1:compile | | +- com.github.luben:zstd-jni:jar:1.5.5-1:runtime | | +- org.lz4:lz4-java:jar:1.8.0:runtime | | - org.xerial.snappy:snappy-java:jar:1.1.10.4:runtime | +- org.immutables:value:jar:2.10.1:compile | +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.15.3:compile | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.3:compile | +- jakarta.validation:jakarta.validation-api:jar:3.0.2:compile | +- org.springframework.security:spring-security-oauth2-client:jar:6.2.1:compile | | +- org.springframework.security:spring-security-oauth2-core:jar:6.2.1:compile | | - com.nimbusds:oauth2-oidc-sdk:jar:9.43.3:compile | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile | | +- com.nimbusds:content-type:jar:2.2:compile | | +- com.nimbusds:lang-tag:jar:1.7:compile | | - com.nimbusds:nimbus-jose-jwt:jar:9.21:compile | +- org.springframework.kafka:spring-kafka:jar:3.1.1:compile | | +- org.springframework:spring-messaging:jar:6.1.3:compile | | +- org.springframework:spring-tx:jar:6.1.3:compile | | - org.springframework.retry:spring-retry:jar:2.0.5:compile | +- io.pivotal.cfenv:java-cfenv:jar:3.1.5:compile | | - com.cedarsoftware:json-io:jar:4.19.1:compile | | - com.novell.ldap:jldap:jar:2009-10-07:compile | +- org.apache.commons:commons-collections4:jar:4.4:compile | +- org.springframework.cloud:spring-cloud-context:jar:4.1.1:compile | - org.apache.commons:commons-text:jar:1.11.0:compile +- com.google.code.findbugs:jsr305:jar:3.0.2:compile +- org.springframework.boot:spring-boot-starter-test:jar:3.2.2:test | +- org.springframework.boot:spring-boot-test:jar:3.2.2:test | +- org.springframework.boot:spring-boot-test-autoconfigure:jar:3.2.2:test | +- com.jayway.jsonpath:json-path:jar:2.9.0:test | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:4.0.1:compile | | - jakarta.activation:jakarta.activation-api:jar:2.1.2:compile | +- net.minidev:json-smart:jar:2.5.0:compile | | - net.minidev:accessors-smart:jar:2.5.0:compile | | - org.ow2.asm:asm:jar:9.4:compile | +- org.assertj:assertj-core:jar:3.24.2:test | | - net.bytebuddy:byte-buddy:jar:1.14.11:test | +- org.awaitility:awaitility:jar:4.2.0:test | +- org.hamcrest:hamcrest:jar:2.2:test | +- org.mockito:mockito-core:jar:5.7.0:test | | +- net.bytebuddy:byte-buddy-agent:jar:1.14.11:test | | - org.objenesis:objenesis:jar:3.3:test | +- org.mockito:mockito-junit-jupiter:jar:5.7.0:test | +- org.skyscreamer:jsonassert:jar:1.5.1:test | | - com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test | - org.xmlunit:xmlunit-core:jar:2.9.1:test +- org.junit.jupiter:junit-jupiter-api:jar:5.10.1:test | +- org.opentest4j:opentest4j:jar:1.3.0:test | +- org.junit.platform:junit-platform-commons:jar:1.10.1:test | - org.apiguardian:apiguardian-api:jar:1.1.2:test +- org.junit.jupiter:junit-jupiter-engine:jar:5.10.1:test | - org.junit.platform:junit-platform-engine:jar:1.10.1:test +- org.junit.jupiter:junit-jupiter-params:jar:5.10.1:test +- org.mockito:mockito-inline:jar:5.2.0:test +- org.junit.jupiter:junit-jupiter:jar:5.10.1:test +- org.junit.vintage:junit-vintage-engine:jar:5.10.1:test | - junit:junit:jar:4.13.2:test +- com.sap.sectesting:fortify-annotations:jar:1.2.2:provided - org.springdoc:springdoc-openapi-starter-webmvc-ui:jar:2.3.0:compile +- org.springdoc:springdoc-openapi-starter-webmvc-api:jar:2.3.0:compile | - org.springdoc:springdoc-openapi-starter-common:jar:2.3.0:compile | - io.swagger.core.v3:swagger-core-jakarta:jar:2.2.19:compile | +- io.swagger.core.v3:swagger-annotations-jakarta:jar:2.2.19:compile | - io.swagger.core.v3:swagger-models-jakarta:jar:2.2.19:compile - org.webjars:swagger-ui:jar:5.10.3:compile
Project Details
Checklist