SIGPROF on perfctr patched kernels

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Use a kernel patched with perfctr 
(http://user.it.uu.se/~mikpe/linux/perfctr/)
2. Run any sample application with the profiler ( CPU and/or heap)
3. The pprof tool shows the Profiling Timer Expired (SIGPROF)

What is the expected output? What do you see instead?
Expecting profile results, but the profile file generated is empty. And 
pprof gives the message "Profiling Timer Expired"

What version of the product are you using? On what operating system?
google-perftools 1.5. 
OS: Linux 2.6.9-78.ELsmp with perfctr patch. RHEL 4.0.7 64-bit

Please provide any additional information below.

Original issue reported on code.google.com by ashwi...@gmail.com on 25 Mar 2010 at 4:53

[GoogleCodeExporter]

I don't know anything about perfctr.  Does it interact with SIGPROF in some way?

} 3. The pprof tool shows the Profiling Timer Expired (SIGPROF)

This happens even for heap profiles?  They don't use SIGPROF at all, so that 
surprises 
me.

Can you show the exact pprof command you ran, and the exact output you got from 
it?

Original comment by csilv...@gmail.com on 25 Mar 2010 at 5:51

• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

I just re-ran profiling, and the heap profiling seems to be generated. Sorry 
about 
that one. 

This is the command I run:

pprof --ps <executable> <profile file> > cpuprofile.ps 
This is the message I get: 
Profiling timer expired pprof --ps <executable> <profile file> > cpuprofile.ps 

The <profile file> is empty. The files are created as per the CPUPROFILE 
environment 
variable, but they have no data in them.

We use the perfctr in conjunction with Performance Application Programming 
Interface 
- http://icl.cs.utk.edu/papi/ - to read out the hardware counters when running 
some 
of the applications. 

Original comment by ashwi...@gmail.com on 26 Mar 2010 at 4:10

[GoogleCodeExporter]

When you say the profile file is empty, do you mean really empty -- like 0 
bytes?  If 
so, my guess is your binary is not shutting down cleanly.  How do you exit the 
program?

} Profiling timer expired pprof --ps <executable> <profile file> > 
cpuprofile.ps 

This means that something is sending a SIGPROF signal to pprof (I had assumed 
the 
SIGPROF was in the application you were running).  Strange.  It's definitely 
not us; 
I don't know who it might be.  Does perfctr send SIGPROF to things?  What 
happens if 
you just run pprof <executable> <profile file>, and play around with it a bit 
in 
interactive mode?

Original comment by csilv...@gmail.com on 26 Mar 2010 at 2:53

[GoogleCodeExporter]

[ashwinks' reply:]

> When you say the profile file is empty, do you mean really empty -- like 0  
> bytes?  If
> so, my guess is your binary is not shutting down cleanly.  How do you exit  
> the
> program?

The program exits normally, and I can see the expected program output.
Yes, the cpu profile files is 0 bytes.

> This means that something is sending a SIGPROF signal to pprof (I had  
> assumed the
> SIGPROF was in the application you were running).  Strange.  It's  
> definitely not us;

Ah ok. I think this may be two different problems. With the perfctr,
the CPU profiling isn't happening.
And, it's also affecting the behaviour of ppprof.

> you just run pprof <executable> <profile file>, and play around with it a  
> bit in
> interactive mode?

I'll try using the interactive mode, and attach the output.

Original comment by csilv...@google.com on 29 Mar 2010 at 5:26

[GoogleCodeExporter]

} Yes, the cpu profile files is 0 bytes.

I think this is the place to start -- that definitely shouldn't be happening.  
How
are you trying to get cpu profiles, just running with the CPUPROFILE environment
variable?  How does your program exit? -- return from main, or calling exit(), 
or
_exit(), or abort(), or some other way?

Do you get the message at program exit that perftools puts out, saying 
something like
PROFILE: interrupts/evictions/bytes = 555/241/21224

If so, what does this line say?

Original comment by csilv...@google.com on 29 Mar 2010 at 5:28

[GoogleCodeExporter]

Any more info on this?

Original comment by csilv...@gmail.com on 7 Jun 2010 at 11:05

[GoogleCodeExporter]

Sorry to get back late on this. 
We don't get any of the interrupts/evictions/bytes. It's just output as 0/0/0. 

We are currently blocking out the perfctr patched kernel machines when we fire 
distributed jobs. 

Original comment by ashwi...@gmail.com on 8 Jun 2010 at 4:06

[GoogleCodeExporter]

OK.  What about the other questions:
---
How
are you trying to get cpu profiles, just running with the CPUPROFILE environment
variable?  How does your program exit? -- return from main, or calling exit(), 
or
_exit(), or abort(), or some other way?
---

Does your binary actually do anything that consumes CPU?  (It's suprising how 
often that turns out to be the problem...)

I'm guessing this has something to do with perfctr -- sounds like that's what 
you're seeing too, since it works when you use non-perfctr kernels -- but I 
don't really know perfctr enough to understand what it might be.  My guess is 
it uses the same mechanisms as the cpu profiler, and they get confused.  Maybe 
itimers don't work with perfctr?

Original comment by csilv...@gmail.com on 8 Jun 2010 at 7:12

[GoogleCodeExporter]

Any more word on this?  Otherwise I'll have to close CannotReproduce.

Original comment by csilv...@gmail.com on 2 Aug 2010 at 4:36

[GoogleCodeExporter]

It's been a year.  Closing.

Original comment by csilv...@gmail.com on 31 Aug 2011 at 11:36

• Changed state: CannotReproduce