Open codeSonic opened 5 years ago
MichaelLogutov
commented
5 years ago Got the same trying to use perfcollect to profile running container. My scenario was like this:
And after Ctrl+C I've got a warning libcoreclr.so not found in perf data. Please verify that your .NET Core process is running and consuming CPU.
And empty trace file.
sywhang
commented
5 years ago Hi @codeSonic,
Could you please provide us with a bit more information? Specifically:
lttng --version)@MichaelLogutov What you are seeing seems to be a different issue... Just to clarify, are you trying to run the perfcollect script inside your Docker container, or are you running it from the host machine? If you're running it from the host machine that's expected because perfcollect won't be able to see that PID inside its own process map. You'd want to run perfcollect from Docker itself. You can check out an example of how to do that in https://github.com/dotnet/corefx-tools/blob/master/src/performance/perfcollect/docker-demo/Dockerfile. If that doesn't solve your issue, could you file a different issue and we can take it there so that we don't have two separate issues going on in the same thread?
Thanks!
paraspatidar
commented
5 years ago Hi ,
Similer issue here.
I have collected trace form Kubernetes container.
when viewing in to container itself by unziping trace and using perf report , it shows all call stack.
however , when i copy same trace zip on windows , it shows events and eventstats but CPU stacks are empty (same as shown by @codeSonic .
here are sample traces : https://drive.google.com/uc?id=1ZM35Cf4WfkckUMigSo0qhwbbdhSEXPBm&export=download
and
https://drive.google.com/file/d/1cg0TnD6VjKcgm89upmZ8Pvn8QW8vIksN/view?usp=sharing
lttng version : lttng (LTTng Trace Control) 2.9.3 - Joannès container OS : PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)"
perf version : perf version 5.0.rc7.g8a61716 perview version : 2.0.26
PS : for dotnet core container , perfcollect install is unable to install perf , thus i manually build perf executable using :
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux/tools/perf
make
cp perf /usr/bin
sywhang
commented
5 years ago Thanks @paraspatidar!
I took a quick look and it seems like this issue may be a duplicate of https://github.com/dotnet/corefx-tools/issues/84. In perfcollect.log, I can see:
Running /usr/bin/perf script -i perf.data.merged -F comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace > perf.data.txt
'trace' not valid for hardware events. Ignoring.
'trace' not valid for software events. Ignoring.
'trace' not valid for hardware-cache events. Ignoring.
'trace' not valid for unknown events. Ignoring.
'trace' not valid for unknown events. Ignoring.
'trace' not valid for unknown events. Ignoring.
Samples for 'cpu-clock' event do not have CPU attribute set. Cannot print 'cpu' field.
Running /usr/bin/perf script -i perf.data.merged -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace > perf.data.txt
Error: Couldn't find script `comm,pid,tid,cpu,time,event,ip,sym,dso,trace'
See perf script -l for available scripts.So in the end perf.data.txt file is not created due to this error and that file is what PerfView reads when it tries to print the stack so you're seeing the empty cpu stacks.
I'll investigate further and get back with a fix soon but just wanted to share this so that your concerns aren't left unanswered :-)
I ran perfcollect from ubuntu VM for asp.net core 2.1 application by specifying pid of the dotnet process. The perf trace file was generated successfully and I was able to view the data on ubuntu with ./perfcollect view session4.trace.zip command. However, when I copied the trace file on windows and opened it in Perfview, it could not load the CPU Stacks. PFB snapshot for the same.
The events data is visible though.
I checked the underlying 'perf.data.txt' file and found it empty, but the 'perf.data' file is not empty.
Can you please help here to see what's going wrong?