scala / scala-jenkins-infra

A Chef cookbook that manages Scala's CI infrastructure.
https://scala-ci.typesafe.com
Apache License 2.0
14 stars 17 forks source link

Enable JVM Servicability Agents (jstack, etc) on slaves #178

Closed retronym closed 8 years ago

retronym commented 8 years ago

https://bugs.openjdk.java.net/browse/JDK-7050524?focusedCommentId=12450509&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12450509

You can revert to the more permissive policy in one of two ways: 1) temporarily (until the next reboot) by changing the value to 0 in /proc/sys/kernel/yama/ptrace_scope, or 2) more permanently by changing the value to 0 in /etc/sysctl.d/10-ptrace.conf and rebooting. Option 1 does not require a reboot and option 2 does require a reboot.

Context: I was working on a script to use in the Jenkins Script Console to list all JVMs running across all slaves and to collect a thread dump from one of them:

import hudson.util.RemotingDiagnostics;

print_ip = 'println InetAddress.localHost.hostAddress';
print_hostname = 'println InetAddress.localHost.canonicalHostName';

jps = "def proc = Runtime.getRuntime().exec(['jps', '-lv'] as String[]); proc.waitFor(); println proc.in.text";

slaveProcess = "jenkins-worker-behemoth-1"
slavePid = "26192"

for (slave in hudson.model.Hudson.instance.slaves) {
  println slave.name;
  println RemotingDiagnostics.executeGroovy(print_ip, slave.getChannel());
  println RemotingDiagnostics.executeGroovy(print_hostname, slave.getChannel());
  println RemotingDiagnostics.executeGroovy(jps, slave.getChannel());
  if (slave.name == slaveProcess) {    
    jstack = "def proc = Runtime.getRuntime().exec(['jstack', '-F', '$slavePid'] as String[]); proc.waitFor(); println proc.err.text";
    println RemotingDiagnostics.executeGroovy(jstack, slave.getChannel());
  }
}

This resulted in:


Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
retronym commented 8 years ago

I can run jstack when ssh-ed to the machine, so my diagnosis was wrong here.