Add support for triggering thread dumps and heap dumps

[gberche-orange]

Following @nebhale 's suggestion to request most needed features in the java buildpack, adding ability to remotely trigger thread dumps and heap dumps would provide lots of value to CF users that can't leverage cloud monitoring solutions such as appdynamics or new relic.

Ideally such dumps could be triggered interactively or programmatically (e.g. by a monitoring robot collecting evidences prior to restarting a specific app instance suspected to be in a deadlock).

The smart atime-based cf files solution implemented by @sparameswaran into weblogic-buildpack seems pretty appealing and could probably be ported over to the java-buildpack and be applicable to all deployment modes (tomcat, java -jar, play ...)

It would be interesting to measure whether a go-lang implementation of the agent would actually save memory usage w.r.t. the current bash-based impl for each script.

[nebhale]

@gberche-orange Sorry for the delay in response, but I've been putting a lot of thought into this issue and how much we want to get involved in this. Personally, I really don't want to get into the business of starting an additional (basically invisible and unmanaged process) inside of the already crowded Warden container. I can see the advantages of doing so, but I'm hoping that there's a better solution out there.

At the moment, my main thinking is that if we can get the multi-port access into containers (ostensibly for debugging) it would give us access via JMX. Given JMX access, all of the dumps that you'd want can be triggered and it wouldn't require a separate process and the file-watching hack. I'm trying to get a read on how likely that functionality is and how long we might have to wait to get it. Until then, I'd like to hold-fire on this issue.

/cc @rmorgan @jbayer

[gberche-orange]

Thanks @nebhale for your study of this suggestion and for the visibility you're trying to get into a future TCP routing feature (latest related vcap-dev@ discussions with james were not precizing timelines).

Would embedding this watch behavior in the JVM address your concern about "an already crowded Warden container" ?

Same mechanism for watching the accessed file modif date from the JVM, the thread dump can them be done using the JVM ThreadMBean and put this dump into stdout for loggregator consumption. This could for example be loaded as a JVM agent in order to keep applicability into a large type of java deployments (java -jar, spring boot, tomcat ...) ?

[nebhale]

Yeah, that's actually quite a good suggestion, albeit one that would fall lower on my list then additional ports. We'll put that in the pot to stew around...

[youngm]

Just an implementation note. This functionality could be easily hooked into a jvm using a javaagent. We do that in a couple of places in our buildpack to set up jmxmp or setup java proxies from http_proxy environment variables, etc.

[nebhale]

@youngm I hadn't thought about it until @gberche-orange. It's certainly the leading candidate that I don't want to do :smile:

[youngm]

@nebhale I agree. This is the best bad solution. :) Good luck pushing for a better one.

[cgfrost]

FYI https://www.pivotaltracker.com/story/show/86779520

[lhotari]

I'm planning to modify the killjava.sh script to upload heapdumps to s3.

I'll add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=some_location to the JVM startup options besides -XX:OnOutOfMemoryError so that there will be a heapdump available when the JVM crashes.

It should be possible to use HeapDumpOnOutOfMemoryError and OnOutOfMemoryError together according to this blog post.

I found a script to do the upload to s3 with curl. A similar approach is used in Heroku's with_jmap script to upload heapdumps to s3.

Heroku has pretty nice options for troubleshooting memory issues in JVM apps: https://devcenter.heroku.com/articles/java-memory-issues

I hope troubleshooting memory issues becomes easier with CloudFoundry Java buildpack. Is there any license restrictions to copy scripts and ideas from Heroku's https://github.com/heroku/heroku-buildpack-jvm-common to CloudFoundry's Java Buildpack or do we have to reinvent the wheel?

[jbayer]

@lhotari perhaps ask the heroku team to add a license to the repo for https://github.com/heroku/heroku-buildpack-jvm-common. the ruby buildpack is MIT for example: https://github.com/heroku/heroku-buildpack-ruby/blob/master/LICENSE

[lhotari]

@youngm @jbayer I've now created a proof-of-concept of the solution that uploads the heapdump to S3 when there is an OOM. It's in the branch of my fork https://github.com/lhotari/java-buildpack/commits/jbp-diagnostics-oom . There is a sample app to test this: https://github.com/lhotari/hello-jbp-diagnostics . The OOMServlet is at /cause-oom path.

This is the example of the output seen in "cf logs appname" when an OOM occurs:

2015-03-04T17:06:33.04-0500 [App/0]      OUT java.lang.OutOfMemoryError: Java heap space
2015-03-04T17:06:33.04-0500 [App/0]      OUT Dumping heap to /home/vcap/app/oom_heapdump.hprof ...
2015-03-04T17:06:41.23-0500 [App/0]      OUT Heap dump file created [727822044 bytes in 8.193 secs]
2015-03-04T17:06:41.25-0500 [App/0]      OUT #
2015-03-04T17:06:41.25-0500 [App/0]      OUT # java.lang.OutOfMemoryError: Java heap space
2015-03-04T17:06:41.25-0500 [App/0]      OUT # -XX:OnOutOfMemoryError="/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"
2015-03-04T17:06:41.25-0500 [App/0]      OUT #   Executing /bin/sh -c "/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh"...
2015-03-04T17:06:41.66-0500 [App/0]      OUT Calculating compressed size first to minimize disk space usage
2015-03-04T17:06:50.73-0500 [App/0]      OUT Compressing and uploading 7649574 bytes to S3. Presigned access url: https://myapp-heapdumps.s3.amazonaws.com/oom_heapdump_1425506801.hprof.gz?AWSAccessKeyId=XXXXXX&Expires=1425679601&Signature=XXXXXXXX
2015-03-04T17:06:59.23-0500 [App/0]      ERR   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2015-03-04T17:06:59.23-0500 [App/0]      ERR                                  Dload  Upload   Total   Spent    Left  Speed
100 7470k    0     0  100 7470k      0  16.9M --:--:-- --:--:-- --:--:-- 18.0M

The heapdump file is compressed with gzip on the fly when it gets uploaded to S3. The gzipping has to be done twice when we don't want to do temp files (since dumps can be large) because Amazon S3 wants to know the Content-Length before the upload starts. You can find the details in the script that contains the bash functions for the upload: https://github.com/lhotari/java-buildpack/blob/jbp-diagnostics-oom/resources/open_jdk_jre/bin/jbp-diagnostics-functions.sh

Besides the OOM heapdump upload script, I have created a solution for requesting heap dumps on demand. That's available from https://github.com/lhotari/java-buildpack-diagnostics-app . I mentioned that in #146 comments.

The primary reason why I created these tools is that I needed a way to debug a slow memory leak in the production app that is hosting grails.org . Now it will be much easier to find the reason of an OOM. Let's see if I can catch the heapdump this time. :)

[lhotari]

CloudFoundry seems to limit disk_quota to 2048 MB . When I set the disk_quota to a higher value, the app won't start and this error message gets returned: The app is invalid: disk_quota too much disk requested (must be less than 2048)

It won't be possible to get heap dumps for JVM apps with large heaps because of this hard limit in disk_quota.

[lhotari]

It looks like the disk_quota limit is configured with the maximum_app_disk_in_mb setting in the cloud controller. The default value is 2048 . Why is the default value so low?

[jbayer]

the default is low to prevent allocation of too much disk. it's configurable.

[gberche-orange]

thanks a lot @lhotari ! This seems great to have both OOMs push heapdumps to S3 (indeed riakCS in our private CF instance case).

The ability to trigger on demand-thread dumps/heap dumps by sending an HTTP request (protected by a secret) to the JVM is pretty useful as well, albeit this makes the assumption the tomcat threads are healthy enough to handle the request and perform a thread/heap dump.

A mechanism external to the JVM (kill -3 or JVM specific IP with jcmd/jstack) would be more resistent to these cases. @nebhale was concerned in https://github.com/cloudfoundry/java-buildpack/issues/70#issuecomment-49409165 on how such "watchdog" process would be monitored, and would add another process within the container process space. May be a top-level process manager could address this. I'm not sure how this would coexist with (future?) cf push -c support.

In terms of out-of-band control flow to send the dump request, we had previously discussed the file-watching hack. @jbayer, as diego seems to allows to bind multiple (HTTP?) ports for the application, could diego transparently allocating a port for diagnostic, similarly to the deprecated DEBUG port in vcap v1 ?

[nebhale]

At the moment we're taking a bit of time on this one to let the Diego changeover get ahead of us. Diego promises to expose both SSH and SCP access to containers giving you the ability to do almost anything (see #146). I believe that this will give users who want it, kill -3, jstack, jcmd, etc. what they want without us baking in logic about S3/raik which many people won't be able to use.

[lhotari]

@jbayer It's configurable when you can control the CF controller. In public cloud services like Pivotal Webservices CF or IBM BlueMix users seem to be limited to 2048 disk space. That's a really big problem in this case (heap dumps) since there isn't enough disk space to write the heap dump to a file at all.

I've been testing the OOM heapdump solution I have in my java-buildpack fork. The problem is that it runs out of diskspace and I cannot catch the heapdump in OOM.

@nebhale Is there plans to provide some support for crash logs (like OOM heap dumps) in CloudFoundry? I don't think ssh/scp access helps when the instance crashes.

I'd like to see some "out-of-the-box" functionality for crash logs in CF. Before we get that, I think the only option is to upload files to some external location like S3. You don't even need a special client for S3 access. The OOM heap dump upload shell script uses curl and openssl to upload to S3. I copied the original solution from this blog post. S3 seems to be the standard for storing binary files over http/https. There are many S3 providers for the public cloud and you get enterprise products for your own datacenter. Why wouldn't many people be able to use it?

The solution I made in my fork is a proof-of-concept. It's not something that can be merged in to the "standard" java-builldpack. I needed the solution now to solve a current production problem and that's why I wrote it. Perhaps something could be reused later on.

Because of the 2GB diskspace limit in CF (in Pivotal WS), I've been playing around with the idea of patching the JVM's sun.jvm.hotspot.utilities.HeapHprofBinWriter class to do gzipping on-the-fly when it's writing the heap dump. Heapdumps seem to compress to about 20% of the original size with gzip and that saves a lot of disk space and band width. It might be even possible to patch HeapHprofBinWriter to upload directly to some S3 endpoint over the network without using any temporary files. :) There are several ways to patch HeapHprofBinWriter. The coolest way would probably be to write a custom JVM agent that uses dynamic bytecode instrumentation to modify HeapHprofBinWriter before it's loaded. It should be quite straight-forward to wrap the FileOutputStream in HeapHprofBinWriter with java.util.zip.GZIPOutputStream by using bytecode instrumentation.

[lhotari]

It looks like the JVM uses a native implementation for heapdumps. I'm not sure why there is the Java version HeapHprofBinWriter available. Patching the native implementation would require a custom JVM build. It will probably just be easier to lift the hard 2048 MB limit in Pivotal WS disk_quota ( maximum_app_disk_in_mb setting in the cloud controller)

[lhotari]

@gberche-orange I was able to find a way to get exactly the same output that jstack gives. It's now done by calling "threadPrint" on DiagnosticCommandMBean in the thread dump servlet. real example of output from this:

2015-03-05 18:30:28
Full thread dump OpenJDK 64-Bit Server VM (25.31-b07 mixed mode):

"http-nio-61519-exec-10" #29 daemon prio=5 os_prio=0 tid=0x0000000002fab800 nid=0x43 waiting on condition [0x00007f4cd749b000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000fe0188b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

I also added servlet for getting memory information. It's useful to have this available before doing a heap dump and that's why it's related to this issue.

/jbp-diagnostics/meminfo example:

JVM memory usage
Heap                                used:   88M committed:  753M max:  753M
Non-Heap                            used:   40M committed:   41M max: 1399M

Memory pools
PS Eden Space                       used:   41M committed:  196M max:  196M
PS Eden Space peak                  used:  196M committed:  196M max:  196M
PS Survivor Space                   used:   32M committed:   32M max:   32M
PS Survivor Space peak              used:   32M committed:   32M max:   32M
PS Old Gen                          used:   13M committed:  524M max:  524M
PS Old Gen peak                     used:   13M committed:  524M max:  524M
Code Cache                          used:    8M committed:    8M max:  245M
Code Cache peak                     used:    8M committed:    8M max:  245M
Metaspace                           used:   28M committed:   29M max:  104M
Metaspace peak                      used:   28M committed:   29M max:  104M
Compressed Class Space              used:    3M committed:    3M max: 1048M
Compressed Class Space peak         used:    3M committed:    3M max: 1048M

The java-buildpack-diagnostics-app now has /jbp-diagnostics/heapdump , /jbp-diagnostics/threaddump and /jbp-diagnostics/meminfo uris for getting diagnostics from the Tomcat instance. It's a limited solution, but it works for my use case of "triggering thread dumps and heap dumps".

There is a simple Java web application to test java-buildpack-diagnostics-app.

[sparameswaran]

The biggest limitation with jmx, servlet or other interactions points is there wont be any response if the jvm starts acting up or thrashing for memory or does not respond. By that time, things can pretty well lock up or gone downhill.

-Sabha

On Thu, Mar 5, 2015 at 10:35 AM, Lari Hotari notifications@github.com wrote:

@gberche-orange https://github.com/gberche-orange I was able to find a way to get exactly the same output that jstack gives. It's now done by calling "threadPrint" on DiagnosticCommandMBean https://github.com/lhotari/java-buildpack-diagnostics-app/blob/cd3158fabc55f8a791eef323c0d1d9c8bfbfdd57/src/main/groovy/io/github/lhotari/jbpdiagnostics/DiagnosticCommandMBeanHelper.java#L14-L24 in the thread dump servlet https://github.com/lhotari/java-buildpack-diagnostics-app/blob/master/src/main/groovy/io/github/lhotari/jbpdiagnostics/ThreadDumpServlet.groovy . real example of output from this:

2015-03-05 18:30:28 Full thread dump OpenJDK 64-Bit Server VM (25.31-b07 mixed mode):

"http-nio-61519-exec-10" #29 daemon prio=5 os_prio=0 tid=0x0000000002fab800 nid=0x43 waiting on condition [0x00007f4cd749b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000fe0188b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)

I also added servlet for getting memory information. It's useful to have this available before doing a heap dump and that's why it's related to this issue.

/jbp-diagnostics/meminfo example:

JVM memory usage Heap used: 88M committed: 753M max: 753M Non-Heap used: 40M committed: 41M max: 1399M

Memory pools PS Eden Space used: 41M committed: 196M max: 196M PS Eden Space peak used: 196M committed: 196M max: 196M PS Survivor Space used: 32M committed: 32M max: 32M PS Survivor Space peak used: 32M committed: 32M max: 32M PS Old Gen used: 13M committed: 524M max: 524M PS Old Gen peak used: 13M committed: 524M max: 524M Code Cache used: 8M committed: 8M max: 245M Code Cache peak used: 8M committed: 8M max: 245M Metaspace used: 28M committed: 29M max: 104M Metaspace peak used: 28M committed: 29M max: 104M Compressed Class Space used: 3M committed: 3M max: 1048M Compressed Class Space peak used: 3M committed: 3M max: 1048M

The java-buildpack-diagnostics-app https://github.com/lhotari/java-buildpack-diagnostics-app now has /jbp-diagnostics/heapdump , /jbp-diagnostics/threaddump and /jbp-diagnostics/meminfo uris for getting diagnostics from the Tomcat instance. It's a limited solution, but it works for my use case of "triggering thread dumps and heap dumps".

The is a simple Java web application to test java-buildpack-diagnostics-app https://github.com/lhotari/hello-jbp-diagnostics.

— Reply to this email directly or view it on GitHub https://github.com/cloudfoundry/java-buildpack/issues/70#issuecomment-77422101 .

[lhotari]

@sparameswaran I agree. it would be nice if CloudFoundry had real support for crash logs. I think it should try to do a heapdump and threaddump before killing the instance when a healthmonitor check fails. These "crash logs" could then be published in some "crash log" storage so that it would be easier to debug why apps crash.

However the biggest limitation currently is the limited disk space (2GB) on the node. In my buildpack, I have changes for triggering an heapdump and uploading to S3 in case of OOM. This currently fails when the heapdump is larger than the available disk space. It's pretty common that heapdumps are in gigabyte size. The OOM heapdump might also timeout if it takes too long. I wonder if it's configurable to let the failing node to run longer before it's killed by the cloud controller? The controlller could just take it out of routing configs so that the node doesn't get more traffic, but just let it live longer so that the crash information could be saved before killing the node. I don't know CF that well that I'd know how this could be implemented. :)

[youngm]

We used to have a crashlog feature but wasn't built for large binaries. It later was the cause of some issues and got deprecated when loggregator came along. Hopefully it gets revisited sometime.

[lhotari]

I found out that IBM BlueMix has their own custom solution for uploading crash logs. https://developer.ibm.com/bluemix/2014/09/15/diagnostic-data-liberty-buildpack/ IBM JVM has it's own custom solution for adding hook scripts to different type of events.

[gberche-orange]

@lhotari just double checking: in https://github.com/cloudfoundry/java-buildpack/issues/70#issuecomment-77394347 you mentionned the failed attempt to patch HeapHprofBinWriter: where you indeed trying to patch execution of the jmap tool, or where you rather asking the jvm performing itself the heapdump as instructed by -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./heapdump.hprof with a modified tools.jar? I'm suspecting the JVM-triggered heapdump could use C native impl, whereas the jmap out-of-process impl to use the java impl.

If this happens to be true, then a bytecode manipulation on tools.jar could modify the behavior of jmap to do the on-the-fly streaming to S3, overcoming the ephemeral FS quota issue.

Concerning the S3 prepreq for the Content-Length could possibly be worked around by:

• using S3 multi-part upload, see http://stackoverflow.com/questions/8653146/can-i-stream-a-file-upload-to-s3-without-a-content-length-header Would be worth checking RiakCS also supports it.
• announcing a guessed larger size (based on a conservative heuristic of compression ratio), and pad the zip/tar.gz file to match the announced size. Maybe an uncompressed tar which contains the gzip heapdump + a second padding file.

[lhotari]

@gberche-orange Thanks for the great implementation ideas for such a tool. I'd like to try doing it, but I currently don't have the time to do it.

For now I've been able to get the dumps with the HeapDumpServlet solution I wrote and it's been good enough for it's purpose I need it for. It's surprisingly fast for dumping heaps from a 1.5 GB JVM heap.

[nebhale]

@gberche-orange Now that we've got Diego (giving you the ability to Debug, JMX, and Profiling), can you think of any additional support you'd like to see here? We're handling OOMs triggering dumps separately in #156 and #161, but I'm not sure if we need any further support for this issue specifically.

[gberche-orange]

thanks ben for the heads up. Diego enables reaching the app instance to trigger the thread dump to ephemeral storage (by providing ability to run commands interactively in app instances - watch to turn off the restart policy when/if this becomes a default - or to ability to local forward traffic to container in order to connect to JMX port properly) .

I'm bit in the hurry these days, but I mean to test triggering heap dump using diego ssh and provide hands feedback on this experience.

As @lhotari pointed out, the remaining challenges might be:

• app ephemeral storage quota might be reached pretty quick
• need to preserve the dump in ephemeral storage from being deleted upon app instance stop (which might be frequent upon low mem conditions)

It might be useful to include a way to automatically upload compressed thread dumps to an S3-compatible object storage. For instance, have the JVM-triggered thread dump be piped to gzip and then to an S3 CLI such as http://randallmcpherson.com/gof3r.html

The java_buildpack could provide a framework to ease this, it could for instance accept S3 credentials bindings.

[lhotari]

@gberche-orange :+1: , gof3r looks like a great solution for streaming large files to s3.

There is a pull request #246 for adding a JVMTI agent that controls the restart of the JVM in the OOM situation. It listens for the JVMTI ResourceExhausted events. It might make sense to add the heap dumping to that agent instead of using -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath options. The agent can be more intelligent about doing the heap dumping so that heap dumps don't get generated if the recent dump was created just a moment ago.

[kelapure]

@lhotari :thumbsup:

It might make sense to add the heap dumping to that agent instead of using -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath options"

This makes complete sense. The responsibility for dealing with the JVM resource exhaustion and followup actions should be owned by one cohesive agent and action.

[lhotari]

For anyone interested, I've created a PR to Spring Boot to add an actuator end point for heap dumps. https://github.com/spring-projects/spring-boot/pull/5670 The first version simply returns a gzipped heap dump. The plan is to later on add features for triggering a heap dump that gets transferred to a remote storage provider like S3 (heap dump transferring could be implemented in application code).

[nebhale]

@gberche-orange Now that Spring Boot has /dump and /heapdump (downloading over HTTP), what are you opinions on this issue? Do you still require buildpack-supplied strategies here?

[gberche-orange]

Thanks @nebhale to resuming this thread. It's great for the spring community be able to leverage builtin http endpoints to trigger thread and heap dumps remotely and stream them gzipped over the network.

I however believe the buildpack/agent support for collecting heap dumps on OOMs, and pushing it to a remote S3 bucket still makes sense because:

• I suspect that upon low memory conditions, the springboot headump support itself would fail to serve the heapdumps over HTTP, whereas a buildpack/agent based implementation might collect the precious dump information (granted there is enough room in garden container in terms of RAM or ephemeral storage if used by the agent).
• AFAIK spring boot heapdump support enables to request a heap dump on demand from an HTTP endpoint, and can not yet be triggered from a JVM-level event such as OOM or other JVMTI ResourceExhausted events: as illustrated by experience from @lhotari, it is common for tricky leaks to take some time or some external factors to reproduce up to an OOM. It is not practical to poll the heapdump endpoint to diagnose such OOM.
• Springboot heapdump support has not yet prioritized support for pushing to a remote S3. Specific impacts to CF users were outlined in https://github.com/spring-projects/spring-boot/pull/5670#issuecomment-215413693
• this will benefit to other JVM-based containers/frameworks in addition to springboot

[michele-mancioppi]

Let me leave this here: https://github.com/SAP/cf-cli-java-plugin

The CLI Java Plugin is a CLI plugin built on top of CF SSH that can create heap and thread-dumps on demand. It does not help in case of container crashes (but that is what the killjava.sh script is for), but it enables simple retrieval also from automated cron jobs.

[nebhale]

@gberche-orange We're now investigating using jvmkill to write heapdumps to a volume_mount. Given that the agent is written in C++, I don't foresee us adding S3 support in the near term, so would this be good enough?

In addition, how do you feel about thread dumps. Is cf ssh and kill -3 enough for that?

[michele-mancioppi]

Hi @nebhale, actually, we are preparing a PR to propose integrating https://github.com/SAP/java-memory-assistant in the Java Buildpack. Early feedback on the idea is very, very welcome :-)

[nebhale]

@michele-mancioppi It's a reasonable contribution, but it specifically cannot replace the behavior on JVM OOM's because it is Java-based. This means that it needs to run within the JVM's restrictions on heap and threads, either of which could already be exhausted by the time it is invoked. One of the reasons for the current jvmkill agent being C++ is that it can then run outside of the exhausted environment and only needs to be concerned with what the container might do to the process as a whole.

[michele-mancioppi]

@nebhale, indeed it is not meant to replace killjvm. It is meant to complete it to take heap dumps in situations that are not yet critical and get them out of the container before the dumps disappear with the latter. We are planning to add more triggers for heap dumps in the future, in particular if there is interest in the community.

[gberche-orange]

Thanks @nebhale for prioritizing this request and sorry for my late response.

Given that the agent is written in C++, I don't foresee us adding S3 support in the near term, so would this be good enough?

This seems a great solution, lowering the preqs for end-users w.r.t. an S3 service. I however see that public CF service providers such as PWS don't seem to yet offer volume services in their marketplace.

How do you environ the user experience to fetch the headdump from the volume_mount for ? Would it be something like the following ?

• he would bind a volume service (such as the https://github.com/cloudfoundry/nfs-volume-release 's broker and test NFS server) to the app (e.g. with a special tag used by the buildpack to identity the volume mount path to use for jvmkill destination).
• the headump would be written to volume_service upon OOMs by the jvmkill agent
• the app team would have to find a way to fetch the heapdump from the volume_service, e.g.
  • bind the volume service app instance to a web ui (such as static buildpack) to serve dump file, as it would be common for NFS server endpoints to not be user-facing (rather private addressing within the container network)

In addition, how do you feel about thread dumps. Is cf ssh and kill -3 enough for that?

Yes I think cf ssf and kill -3 is sufficient for user-triggered thread dumps, in particular given springboot allows for triggering thread dump from http actuator endpoints.

However, it might make sense for jvmkill to trigger a thread dump as well on OOMs, potentially with an opt-out configuration flag in the buildpack.

[nebhale]

@gberche-orange To be honest, I'm a little unclear as to weather volume services will ever come to PWS (and its ilk), but I'm definitely viewing the new jvmkill functionality as rarely used (I think the histogram is going to go a long way) and primarily used internally by enterprises.

As has been related to me by the team working on volume services, they think that most companies will expose an existing NFS server that they already have using volume services (you can put in existing credentials when creating the service instance). I'd have expected that this existing NFS system would already have other ways of accessing it (mount it into your OS or to another server somewhere) and so the files would already be available. However, if that's not true then use some sort of front end would need to be placed over it. Your idea of a static application that just serves the contents of the services mount seems like a reasonable choice, and maybe something that we want to codify into a real project somewhere. Maybe we should put together a meeting to talk about this with the Volume Services team a bit; it seems like your requirements are different than they might expect.

Let me give some thought about thread dumps on OOM.

[nebhale]

@gberche-orange I'm going to close this issue out as complete against heap dumps. I've opened a new issue (#451) to address thread dumps separately.

[gberche-orange]

Thanks a lot @nebhale for implementing this feature in https://github.com/cloudfoundry/java-buildpack/commit/c7148c3be2d75dc892818e173433fa51ff6e7acf and documenting it as part of the standard JREs

The volume service enables operators to leverage NFS or other potential storage backends (say distributed block storage). The java buildpack heapdump to mounted volumes seems to me a useful support for application teams in the enterprise to accessing heap dumps.

For the cases where the volume service is not accessible to app developers (e.g. in my company production NFS mounts are not available on the same networks than the CC API or application domains and usually can't be easily mounted on desktop machines), and given that for now all volume services are shared, as a the last resort, a 2nd cf app bound to the volume service can provide listing and downloading of heapdump files. A static buildpack could be used as previously mentionned, or any more sophisticated file manager UI/WebDav server (such as https://owncloud.org/features/)