context not found with large number of threads

[GoogleCodeExporter]

What steps will reproduce the problem?

Running the attached code fails with (roughly) 25 or more threads.

Eg. 
> python y.py 50
[*]     [yappi-err]     context not found.
[*]     [yappi-err]     context not found.
[*]     [yappi-err]     context not found.
[*]     [yappi-err]     context not found. 
...

What is the expected output? What do you see instead?
> python y.py 5
name                                 #n       tsub       ttot       tavg
../python2.6/threading.py.__init__:5 20       0.000035   0.000035   0.000002
..lib/python2.6/threading.py._note:6 20       0.000028   0.000028   0.000001
..ython2.6/threading.py._is_owned:22 10       0.000044   0.000088   0.000009
...

What version of the product are you using? On what operating system?
Python 2.6.5, Ubuntu Linux, yappi-0.52.zip

Please provide any additional information below.
A similar script using thread.start_new_thread() instead of inheriting from 
threading.Thread doesn't seem to have the problem.

Original issue reported on code.google.com by dribna...@gmail.com on 25 Feb 2011 at 3:08

[GoogleCodeExporter]

Cannot see the included file y.py. Can you please re-send the attachment?

Original comment by sum...@gmail.com on 27 Feb 2011 at 9:45

[GoogleCodeExporter]

A copy of the problem with some example source code is at:
http://www.mail-archive.com/python-list@python.org/msg306045.html

Not sure why the attachment didn't work. I've re-written the code out here:

import yappi
import time
import threading
import sys

class MyThread(threading.Thread):
    def run(self):
        time.sleep(1)

n = int(sys.argv[1])

yappi.start()
for i in range(0,n):
    c = MyThread()
    c.start()
time.sleep(1)
yappi.print_stats()
yappi.stop()

Original comment by dribna...@gmail.com on 27 Feb 2011 at 10:14

[GoogleCodeExporter]

Hello,

I think the problem is fixed. Can you also please confirm the problem is fixed 
in your setup and stat outputs are valid? You can find the development source 
here: https://bitbucket.org/sumerc/yappi/

Thanks,

Original comment by sum...@gmail.com on 28 Feb 2011 at 9:17

[GoogleCodeExporter]

This seems to work on the basic example above. However, when I run a heavily 
threaded application I'm getting lots of the following warnings printed out:
[*] [yappi-err] Context add failed. Already added?(0x21d98c0, 139712695199488)
[*] [yappi-err] Context add failed. Already added?(0x20df110, 139712720377600)
[*] [yappi-err] Context add failed. Already added?(0x251a360, 139712711984896)
[*] [yappi-err] Context add failed. Already added?(0x22b5080, 139712695199488)

It seems to complete successfully though.

Is this likely to be an issue or can they safely be ignored?

Original comment by dribna...@gmail.com on 28 Feb 2011 at 11:06

[GoogleCodeExporter]

Very good inspection. This is an issue I expected. This can safely be ignored, 
however I need to make sure this error message only happens when a specific 
compile option is provided.  Just to make sure, can you confirm the output is 
valid in the second example with heavy multithreaded app.?

Original comment by sum...@gmail.com on 28 Feb 2011 at 11:46

[GoogleCodeExporter]

it seems to be working fine now if I ignore the warnings. The statistics 
generated seem reasonable for the application as well. I was getting some 
strange timings but I think they're all to do with blocking IO cases 
(http://code.google.com/p/yappi/wiki/multithreaded_statistics) rather than an 
error being introduced by this patch.

Original comment by dribna...@gmail.com on 28 Feb 2011 at 4:57

[GoogleCodeExporter]

Accurate timing statistics with heavy I/O bound multithreaded applications are 
currently not possible (as explained in the Wiki page you suggested.) But I am 
thinking to get help from OS for specific platforms on future versions of yappi 
for these situations. Anyway, thanks for your help, will release 0.53 ASAP.

Original comment by sum...@gmail.com on 1 Mar 2011 at 2:08

[GoogleCodeExporter]

fixed in v0.53.

Original comment by sum...@gmail.com on 3 Mar 2011 at 12:52

• Changed state: Fixed