CPU profiler only works on main thread on linux in multi-threaded programs

[GoogleCodeExporter]

What steps will reproduce the problem?
1.Write a simple multi-threaded program using pthreads
2.Enable multi-threaded CPU profiling
3.Use ProfilerStart/ProfilerStop API to start and stop profiling.

What is the expected output? What do you see instead?
I expected to see all threads profiled. Only main thread gets profiled.

What version of the product are you using? On what operating system?
OS:     RedHat on x86_64 (amd)
Kernel: 2.6.9-89.ELsmp
google-perftools: 1.6

Please provide any additional information below.
Found two workarounds, but none of them work for me. I can't change source code.
Workarounds:
1. Call ProfilerRegisterThread() in each thread function.
2. Create a separate thread to send SIGPROF to other threads we want profile  
(Please see details in test case) 

Has anybody else faced the same issue?

Original issue reported on code.google.com by pavl...@gmail.com on 19 May 2011 at 6:46

Attachments:

• TestCase.cpp

[GoogleCodeExporter]

We know that older versions of the linux kernel can only profile the main 
thread (this is documented in profiler.h but should probably be more 
prominent).  Our belief was 'older versions' is linux 2.4 and before.  However, 
maybe it's also 2.6.9 somehow?

In any case, this definitely seems to be a kernel issue, and not something that 
we can easily resolve at the application layer.

Original comment by csilv...@gmail.com on 20 May 2011 at 12:21

• Added labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Haven't heard anything in a while, so closing.  Feel free to reopen if (when) 
you have more info! 

Original comment by csilv...@gmail.com on 18 Oct 2011 at 6:18

• Changed state: CannotReproduce

[GoogleCodeExporter]

This reproduces for me with Ubuntu 13.10 for x86_64. Whether or not you fix 
this I would have saved at least half of a day if you updated your docs (wiki 
page, profiler.h) to indicate that this might be a problem.

installed libgoogle-perftools-dev and google-perftools via (apt-get install 
$package-name). This gives me gperftools 2.1...

$ apt-cache showpkg libgoogle-perftools-dev
Package: libgoogle-perftools-dev
Versions: 
2.1-1 
(/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_saucy_main_binary-amd64_P
ackages) (/var/lib/dpkg/status)

The above depend on libunwind...
$ apt-cache showpkg libunwind8-dev
Package: libunwind8-dev
Versions: 
1.1-2ubuntu3 
(/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_saucy_main_binary-amd64_P
ackages) (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_saucy_main_binary-amd64_Packages
                  MD5: 968fa5fbcdf32543dedc4d9d40405c26
 Description Language: en
                 File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_saucy_main_i18n_Translation-en
                  MD5: 968fa5fbcdf32543dedc4d9d40405c26

Reverse Depends: 
  libunwind8-dev:i386,libunwind8-dev
  libunwind-setjmp0-dev,libunwind8-dev 1.1-2ubuntu3
  libgoogle-perftools-dev,libunwind8-dev
Dependencies: 
1.1-2ubuntu3 - libunwind8 (5 1.1-2ubuntu3) libunwind1-dev (0 (null)) 
libunwind1-dev:i386 (0 (null)) libunwind7-dev (0 (null)) libunwind7-dev:i386 (0 
(null)) libunwind7-dev (0 (null)) libunwind7-dev:i386 (0 (null)) 
libunwind8-dev:i386 (0 (null)) 
Provides: 
1.1-2ubuntu3 - 

Host has gcc 4.8.1 and glibc is...
GNU C Library (Ubuntu EGLIBC 2.17-93ubuntu4) stable release version 2.17, by 
Roland McGrath et al.
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.7.3.
Compiled on a Linux 3.11.3 system on 2013-10-12.
Available extensions:
    crypt add-on version 2.1 by Michael Glad and others
    GNU Libidn by Simon Josefsson
    Native POSIX Threads Library by Ulrich Drepper et al
    BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/eglibc/+bugs>.

I was trying to profile mongodb and that can be built with an option to link 
with libprofiler (scons --use-cpu-profiler). When using mongodb with that 
option the profiler output file was always empty. Only after adding calls to 
ProfilerRegisterThread as shown below was there useful data in the profiler 
output file.

        static void* handleIncomingMsg(void* arg) {
            TicketHolderReleaser connTicketReleaser( &Listener::globalTicketHolder );
            ::ProfilerRegisterThread();
            log() << "profiler registered thread" << endl;

Original comment by mdcal...@gmail.com on 22 Apr 2014 at 3:40

[GoogleCodeExporter]

I'm pretty sure it (having to register every thread) is applicable to any 
recent linux. Seen this myself (see https://github.com/alk/gperf-all-threads).

I was thinking it is documented, but apparently it's not.

I'm not sure at this time, if we can fix it or not, but at least we need to 
document it.

Therefore I believe it's best to reopen it.

Original comment by alkondratenko on 22 Apr 2014 at 7:06

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by alkondratenko on 22 Apr 2014 at 7:07

• Changed title: CPU profiler only works on main thread on linux in multi-threaded programs

[GoogleCodeExporter]

Forgot to include this from the reproduction details

$ uname -a
Linux mark-uvm 3.11.0-19-generic #33-Ubuntu SMP Tue Mar 11 18:48:34 UTC 2014 
x86_64 x86_64 x86_64 GNU/Linux

Original comment by mdcal...@gmail.com on 22 Apr 2014 at 8:23

[GoogleCodeExporter]

Actually my statement above is wrong. Any recent Linux does have per process 
profiling clock and not per-thread. So it should profile all threads 
automatically. If sometimes profile is empty then there's some other bug. I'm 
not sure yet what it might be.

Original comment by alkondratenko on 5 Aug 2014 at 6:24

[GoogleCodeExporter]

Original bug was occurring on 2.6.9 which is before linux got per-process 
profiling timer. So original bug was indeed limitation of kernel. And indeed on 
recent kernel I am seeing thread functions in profile.

I'll separate mongodb issue to different ticket later. And I'm sure I saw 
something similar myself on couchbase-server which is indeed very weird.

Original comment by alkondratenko on 5 Aug 2014 at 6:33

[GoogleCodeExporter]

I was unable to reproduce the bug on recent kernel. Even on 13.10. I suspect 
the problem of empty profile could be caused by unclean shutdown of server 
process in which case profiler did not flush samples to file. I cannot explain 
why registering threads helped.

If you can still reproduce the problem please file new bug and add as much 
details as possible.

Original comment by alkondratenko on 11 Aug 2014 at 2:05

• Changed state: CannotReproduce