On some of our build relying on sbt-assembly to create shaded jar we have observed big usage of native memory.
For a 6G max heap JVM, the pod memory usage is above 12G
Analysing the native usage with libtcmalloc_and_profiler.so.4 and google-pprof we observed that the native usage is due to native usage of Inflater used to unzip jar:
Current code keep all jar file handler which leads to all native inflater being kept with all the memory those are using
In order to avoid this, the proposal is to load all entry data from each jar as ByteArrayInputStream instead of keeping the stream and immediately close the jar once all entries are loaded
Here the native memory usage after this change
The counterpart is an increase of the heap usage.
On our specific build we had to increase heap size to 7G but pod memory usage has decreased below 10G:
Issue description
On some of our build relying on
sbt-assemblyto create shaded jar we have observed big usage of native memory. For a 6G max heap JVM, the pod memory usage is above 12GAnalysing the native usage with
libtcmalloc_and_profiler.so.4andgoogle-pprofwe observed that the native usage is due to native usage of Inflater used to unzip jar:Src code ref of the native inflater code: Java_java_util_zip_Inflater_init and Java_java_util_zip_Inflater_inflateBytesBytes
Proposal
Current code keep all jar file handler which leads to all native inflater being kept with all the memory those are using
In order to avoid this, the proposal is to load all entry data from each jar as ByteArrayInputStream instead of keeping the stream and immediately close the jar once all entries are loaded
Here the native memory usage after this change
The counterpart is an increase of the heap usage. On our specific build we had to increase heap size to 7G but pod memory usage has decreased below 10G: