Open alicegoldfuss opened 7 years ago
Looks like it's only creating the .data file and not .stacks or .collapsed
perf-map-agent]# ls -la /tmp/ | grep 161991
-rw------- 1 root root 1919352 Jan 24 19:40 perf-161991.data
Can you try without -- sleep 30
? perf-java-record-stack
already does -- sleep $PERF_RECORD_SECONDS
.
I see that this is not particularly well documented...
You could also try enabling set -x
in one or several of the scripts to see where it exits. (Shell scripting is not my expertise so any help is appreciated.)
I'm getting the same results without sleep 30
and even a vanilla run like
$ ./bin/perf-java-flames 161991
Where is the resulting svg file supposed to turn up?
In the same directory as the script is run but it should show a line with the name.
Yeah it's definitely not showing up there. I'm going to try to create a Java FlameGraph with the manual steps.
To eliminate some suspects:
bash: sudo: java: command not found
even when run with root with java in the path1.8.0_102-b14
perf
and the FlameGraph
repo, just not with any of your tools.Thanks! I managed to reproduce this issue locally and have a fix. I'll send a PR in a second, but it's very minor so if you can't wait you can go ahead and fix locally by applying the following:
diff --git a/bin/create-java-perf-map.sh b/bin/create-java-perf-map.sh
index 52ee75d..b297067 100755
--- a/bin/create-java-perf-map.sh
+++ b/bin/create-java-perf-map.sh
@@ -24,5 +24,5 @@ fi
[ -d "$JAVA_HOME" ] || (echo "JAVA_HOME directory at '$JAVA_HOME' does not exist." && false)
sudo rm $PERF_MAP_FILE -f
-(cd $PERF_MAP_DIR/out && sudo -u \#$TARGET_UID java -cp $ATTACH_JAR_PATH:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID "$OPTIONS")
+(cd $PERF_MAP_DIR/out && sudo -u \#$TARGET_UID $JAVA_HOME/bin/java -cp $ATTACH_JAR_PATH:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID "$OPTIONS")
sudo chown root:root $PERF_MAP_FILE
Which just carries the JAVA_HOME through to the sudo command
See PR #51
Please let me know if the fix helps your ultimate goal which is to get flame graphs. Also note that if you are aiming to collect machine wide stats for many Java processes @brendangregg has the jmaps scrips which creates a map file for all java processes which can be used as part of producing machine wide profile: https://github.com/brendangregg/FlameGraph/blob/3da963a74a686e2caea489ba637f6afdb6d6658a/jmaps
I suspect this fix will still fail for me, due to a bug in Java that requires me to dump the symbols as the user of the running Java process. But I will let you know!
Also thanks for the jmaps link, but there's only one Java process on this machine.
I added the fix but it still fails, even when running as root. New error though:
# ./bin/perf-java-top 161991
sudo: unable to execute /home/alice/jdk1.8.0_102/bin/java: Permission denied
# ls -la /home/alice/jdk1.8.0_102/bin/java
-rwxr-xr-x 1 alice alice 7734 Jan 24 00:33 /home/alice/jdk1.8.0_102/bin/java
I get the same error when running as alice
. And yes I can run java directly by calling that path.
OK... Not seen that one before. For what it's worth my java executable has the exact same permissions. Running the script as root works for me if I setup the JAVA_HOME environment variable.
We can work out an alternative, I think. The permissions game in the scripts is around 2 files, the map file and the perf.data file:
echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid
This will allow users to collect perf data for running processes.
Using the above I've setup boxes where users are allowed to perf profile their own Java processes with slightly modified scripts, essentially removing the sudo prefix everywhere and the file ownership manipulation. I've not attempted to merge these efforts back. Reflecting on this, perhaps the issue you are seeing is because the user whose process you are trying to profile does not have permission to run you Java installation? Maybe pointing JAVA_HOME at an installation available to all users will solve the issue?
Some security enforcement preventing alice from executing things? like seccomp?
FWIW, my jmaps tool also works around the perf issue of needing the /tmp/perf*map files as owned by root.
Apologies for going dark. I've been digging into this issue with the manual commands.
The issue comes down to containers and namespacing. The Java process I'm trying to profile is running inside a container. The process is owned by a user inside the container, but only has a UID exposed to the host. Even spoofing a user with that UID on the host doesn't work when trying to dump symbols. And I can't do the profiling inside the container, because perf isn't installed and the version of Ubuntu running inside the container is too new for the underlying host kernel to have a supported perf package.
My planned workaround (which I haven't verified, but I believe will work) is:
perf-pid.map
onto the underlying host via a mounted volume.perf-pid.map
filename to match the Java process PID as seen by the host (and chown
to root).perf
and FlameGraph scripts on the host, using the renamed perf-pid.map
file.I think this will give me what I want.
@alicegoldfuss Thanks for sharing your use case in more detail. I bow to your Linux Fu powers, sounds like you are on your way to cracking it, when you do please share the details. It would perhaps help to add this to the wiki. I have not looked much into it, but this project: https://github.com/chbatey/docker-jvm-flamegraphs By @chbatey aims to demo a solution to what seems like a similar challenge.
And the relevant blog post to go with the repo: http://batey.info/docker-jvm-flamegraphs.html
Ah, looks like this person has come to the same conclusion as me! That's comforting :)
@alicegoldfuss if nothing else I at least hope I've introduced you to the right person :-)
Ah, right, containers and perf. I've been meaning to post a blog post too -- we've all probably been working on the same problem. :)
Christopher's post is good, but he needs to let the JVM warm up a bit more -- too many "Interpreter" frames -- they haven't hit CompileThreshold yet.
Ah, this is about containers. I also tried to get it working but only half-hearted. I would also be interested in getting this to work. Thanks for having the discussion here and the extra links, @alicegoldfuss, @nitsanw, and @brendangregg.
My workaround worked!
I'm going to dance to something and then document what I did.
Turned it into a blog post. Thanks everyone for your help: http://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/
Thanks a lot, @alicegoldfuss for documenting your findings!
I added transparent support for containers in jmaps
: https://github.com/brendangregg/FlameGraph/pull/171.
To add to the "Why?" section of blog post by @alicegoldfuss, the reason seems to be this:
// "/tmp" is used as a global well-known location for the files
// .java_pid<pid>. and .attach_pid<pid>. It is important that this
// location is the same for all processes, otherwise the tools
// will not be able to find all Hotspot processes.
// Any changes to this needs to be synchronized with HotSpot.
It’s a little unfortunate that this solution is Docker-specific (relies on docker exec). Perhaps we can make it more general by using nsenter?
On Wed, May 16, 2018 at 08:05 Ivan Babrou notifications@github.com wrote:
I added transparent support for containers in jmaps: brendangregg/FlameGraph#171 https://github.com/brendangregg/FlameGraph/pull/171.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jvm-profiling-tools/perf-map-agent/issues/50#issuecomment-389395356, or mute the thread https://github.com/notifications/unsubscribe-auth/ABm3vVdpi3fu_4QJGMKjWaEv31CJijXrks5ty7N-gaJpZM4Lsrjd .
@goldshtn I replied to your comment in the PR. Let's have PR related comments there.
I managed to run the attach script from the host namespace. I haven't properly integrated that as it needs some hacking of internals from tools.jar because as you say above the attach mechanism relies on well-known paths shared between attach and target JVM. Right now it will only work if the target process has PID 1 in the container.
See jvm-profiling-tools/perf-map-agent/compare/jr/attach-to-container-from-host
I'm trying to create a Java process FlameGraph with
perf-java-flames
. It seems to run successfully, but I can't find the resulting svg file.CentOS 7 3.10.0-327.36.3.el7.x86_64 cmake version 2.8.12.2
Up-to-date versions of perf-map-agent and the FlameGraph repo.