search

cloudfoundry / cf-java-client

Java Client Library for Cloud Foundry
Apache License 2.0
328 stars 318 forks source link

ResourceMatchingUtils - OutOfMemoryError #1098

Closed shalomyasap closed 3 years ago

shalomyasap commented 3 years ago

Hello, We are using Spring Cloud DataFlow (V2.7.1) with skipper server (V2.6.1) at CF environment. Skipper server set with 16GB. Attempt to upgrade stream the streams fails with OOM exception. From stack trace its seems there is bug at ResourceMatchingUtils. getArtifactMetadataFromZip, line ~88.

Restarting the skipper solve this issue but this will reproduce after few attempts to upgrade the stream again.

Attaching stack trace.

Will appreciate your assistance. Regards, Shalom ResourceMatchingUtils - OutOfMemoryError.log

shalomyasap commented 3 years ago

Hello, any update regarding this issue? Ill appreciate if someone can take a look since this issue reproduced on regularly base so I think this is critical on Cloud Foundry.

Thanks, Shalom

dmikusa commented 3 years ago

Hi, sorry for the delayed response.

I took a quick look at this. You're seeing a java.lang.OutOfMemoryError: Direct buffer memory, which is unlikely to be from the cf-java-client itself, but dependencies we use. The cf-java-client isn't using native memory directly, but that particular call is triggering some NIO operations, which can use direct memory. In particular, it's reading from the zip file here -> https://github.com/cloudfoundry/cf-java-client/blob/main/cloudfoundry-util/src/main/java/org/cloudfoundry/util/FileUtils.java#L140, which is the line causing direct buffer memory to be allocated.

My suggestion would be to increase the size of the direct memory available to your app. By default, the Java buildpack and it's memory calculator allocate a pretty small amount of direct memory, 10M. Try going larger like 20 or 30M and see if that helps. If the problem goes away, great. It was just undersized. If it still happens, but just takes longer then maybe there's a leak. If that happens, we'd have to capture some memory usage metrics & info to investigate more.

Either case, please give this a try and report back on how it goes. Thanks!

dmikusa commented 3 years ago

Closing as there has been no update. If the suggestion above hasn't helped, please report your feedback and reopen this issue. Thanks