Closed chefhoobajoob closed 4 years ago
I've found that if I explicitly hardwire the settings below, I can get a successful launch. I can't remove either setting or the application won't start. I do not want to have to explicitly set values for these, but I am unable to find an alternative configuration for the memory calculator that works, and jvmkill
did not help in determining which of these settings would need to change - I had to randomly poke around with experimental values to find these.
JAVA_OPTS: '-XX:MaxMetaspaceSize=512M -XX:ReservedCodeCacheSize=512M'
The information exposed by jvmkill
is simply what the JVM exposes to JVMTI agents. When that agent receives a resource exhausted notification, it dumps as much as it can but cannot offer any further diagnostic information beyond that.
I'd be really surprised is the ReservedCodeCache
is the resource that's being exhausted as it's a JIT cache that is evicted when more space is needed. Metaspace is a likely culprit though, especially if you're using nashorn and dynamically creating classes at runtime that would live in metaspace.
The memory calculator is the result of analysis of thousands of real-life applications and while its algorithm is good for most applications, it has a particular blindspot for applications that dynamically create classes (Groovy applications are significantly affected by this too). Because metaspace can only be determined by knowing the number of classes that will be loaded at runtime, if the classes don't exist on the filesystem before startup, then we can't use them in the calculation. Your choice to explicitly set the value for that memory region seems to be the right one for your application.
Thanks @nebhale - we do run javascript on nashorn together with vanilla java classes in these services. I’m convinced you’re right that this blind spot is exactly what we ran into. We stopped processing JAVA_OPTS altogether for this reason - that was the fastest path back to stability.
I only recently learned that the java buildpack was feeding memory settings to apps via
JAVA_OPTS
(our stack doesn't know anything aboutJAVA_OPTS
), so my app's been successfully launching in the meantime using whatever defaults the JVM uses, no explicit memory configuration whatsoever.Now that I've incorporated
JAVA_OPTS
into launch scripts, the jvm is picking up those settings plus thejvmkill-1.16.0_RELEASE
agent, and I'm trying to come up to speed on how the memory configuration and jvmkill behavior is supposed to work, because now startup is consistently failing.When app start fails, this is all I see in the logs:
Followed by memory and thread dump data, followed by:
I've been pushing memory configuration extremely high, way past what's reasonably required, to see if I can find even a single configuration that won't get killed by
jvmkill
, but so far no luck, I'm not finding any clues from either the memory or thread dumps, andjvmkill
itself does not log any rationale for its decisions.For example, for one recent attempt, the configured app memory was:
The memory calculator configuration was:
At startup, this setup results in this memory configuration:
For this startup sequence, I have the
JAVA_OPTS
value mapped over toJAVA_TOOL_OPTIONS
, which results in this message from the jvm:It runs for a bit until
jvmkill
kills it:...with these memory stats:
...and a thread dump for these threads:
Nothing in any of these diagnostic data looks suspicious: heap utilization is ~6% of the max, metaspace is ~75% of the max, class space is below capacity, Compressed Class Space is well below the max, there are no code cache warnings, thread count is about right. The largest thread stacks are coming from Nashorn, with a depth range of 160-199, but it seems quite unlikely that either of those is exceeding 4M.
Anyway, I can't tell from any of this what
jvmkill
is reacting to - am I missing something obvious? Can I getjvmkill
to be more verbose about what it's reacting to?