segmentation fault

[GoogleCodeExporter]

I'm trying to profile an application, but allways got a segmentation fault.

Running in gdb, I got this:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7efb8d0 (LWP 26690)]
hadd (ht=0x8832a84, key=143497780, val=1) at _yhashtab.c:41
41              next = p->next;
(gdb) bt
#0  hadd (ht=0x8832a84, key=143497780, val=1) at _yhashtab.c:41
#1  0xb7b6fa0d in spush (cs=0x87d7d8c, ckey=0x88d9a34) at _ycallstack.c:93
#2  0xb7b6ee3b in _call_enter (self=<value optimized out>, frame=0x882e384,
arg=0x81474d8)
    at _yappi.c:271
#3  0xb7b6f479 in _yapp_callback (self=0x0, frame=0x882e384, what=0,
arg=0x81474d8) at _yappi.c:350
#4  0x080c76fe in ?? ()
#5  0x00000000 in ?? ()

... imports ...

import yappi
yappi.start()

Application().run()

yappi.stop()

for it in yappi.get_stats(yappi.SORTTYPE_TTOTAL, yappi.SORTORDER_ASCENDING,
yappi.SHOW_ALL):
    print it

Deleting the last lines makes the application work without segmentation
fault, but naturally without printing the profile information:

... imports ...

import yappi
yappi.start()

Application().run()

Original issue reported on code.google.com by mjan...@gmail.com on 20 Feb 2010 at 12:21

[GoogleCodeExporter]

I got the segmentation fault on Debian Lenny, with python 2.5.4 and 
yappi-0.3beta.

Original comment by mjan...@gmail.com on 20 Feb 2010 at 12:28

[GoogleCodeExporter]

yappi-0.2beta works properly

Original comment by mjan...@gmail.com on 20 Feb 2010 at 12:50

[GoogleCodeExporter]

Working on it.

Original comment by oktaka...@gmail.com on 21 Feb 2010 at 12:01

[GoogleCodeExporter]

Original comment by sum...@gmail.com on 21 Feb 2010 at 12:19

• Changed state: Started

[GoogleCodeExporter]

Both of the SEG faults indicate some kind of heap corruption. Can you please 
answer
following:

1) How many threads is your application creating?
2) Does your app crash with the following flow?
import yappi
yappi.start()
Application().run()
yappi.stop()

3) Does your app crash with the following flow?
import yappi
yappi.start()
Application().run()
yappi.stop()
yappi.print_stats()

4) Does your app crash with the following flow?
import yappi
yappi.start()
Application().run()
yappi.print_stats()

Original comment by sum...@gmail.com on 21 Feb 2010 at 2:31

[GoogleCodeExporter]

The app creates 13 threads.

Crashes both in 2, 3 and 4 cases. I attached the back traces I got when compiled
yappi with extra_compile_args = ["-D DEBUG_CALL"] in setup.py.

Original comment by mjan...@gmail.com on 21 Feb 2010 at 10:14

Attachments:

• 1.txt
• 2.txt
• 3.txt

[GoogleCodeExporter]

Can you confirm below is working fine as you previously stated? 

import yappi
yappi.start()
Application().run()

Original comment by sum...@gmail.com on 21 Feb 2010 at 1:39

[GoogleCodeExporter]

And also, are you using a Big Endian Debian Lenny distro?

Original comment by oktaka...@gmail.com on 21 Feb 2010 at 2:05

[GoogleCodeExporter]

import yappi
yappi.start()
Application().run()

I tried this again, and worked only once out of 10 tests. Python can't allways 
print
back trace when it fails. A back trace attached.

I'm using the x86 (32bit) version of Debian Lenny 5.0.4 on a Core2Duo E4400 CPU.

Original comment by mjan...@gmail.com on 21 Feb 2010 at 7:47

Attachments:

• 1.txt

[GoogleCodeExporter]

Hi mjanos,

With the statement that your app is working fine on v0.2 seems there is some 
kind of
heap corruption on your system with the internal hash table of yappi v0.3. I 
cannot
be able to reproduce the problem since it has sporadic behavior( sometimes 
double
free, sometimes mem corruption). To narrow down the problem, I am suggesting to
replace the files in the v0.3 _yhashtab and _ymem, and debug the problem from 
there.
I have prepared two files which will happily compile with the v0.3 once copied 
to its
directory. Please help me with the following steps:

1) Copy _yhashtab.c, _yhashtab.h to the yappi v0.3 directory, overwrite or 
rename the
existing.
2) Compile yappi again, run your application. 
3) If problem persists, please copy _ymem.c to the directory again, and compile
re-run your app.

Of course, GDB traceback would be great if possible. 

Thanks again,

Original comment by sum...@gmail.com on 21 Feb 2010 at 11:30

Attachments:

• _yhashtab.c
• _ymem.c
• _ymem.c

[GoogleCodeExporter]

Hi,

I replaced the _yhashtab.c file (I think you forgot to upload _yhashtab.h) and
reinstalled yappi.

1)
import yappi
yappi.start()
Application().run()

In this case, the app worked properly (I ran it about 10 times).

2)
import yappi
yappi.start()
Application().run()
yappi.stop()

In this case, the app worked properly (I ran it about 10 times).

3)
import yappi
yappi.start()
Application().run()
yappi.stop()
yappi.print_stats()

In this case, the app crashes all the time with the python back traces attached 
in
3_1.txt and 3_2.txt, and sometimes only with Segmentation fault message.

4)
import yappi
yappi.start()
Application().run()
yappi.print_stats()

Crashed with similar messages like in 3).

I tried to run it in gdb, see 4_1.txt and 4_2.txt.
I tried to run it in gdb with python-dbg (in the hope we get debug symbols for 
bt),
see 4_python-dbg.txt.

Original comment by mjan...@gmail.com on 22 Feb 2010 at 7:26

Attachments:

• 3_1.txt
• 3_2.txt
• 4_1.txt
• 4_2.txt
• 4_python-dbg.txt

[GoogleCodeExporter]

I replaced the _ymem.c too, I got similar errors like in 4).

Original comment by mjan...@gmail.com on 22 Feb 2010 at 7:33

[GoogleCodeExporter]

Thanks for all your help mjanos. I am suspecting that you have some linkage 
problems
yesterday for some reasons(files not changes did not link with the new 
versions, it
was not a clean build, compilation ended up having previous files using new 
linked
files, this happens to me sometimes). I have prepared two emergency 
distributions for
you. Can you please help me with the following?

1) Extract yappi.03.h1.rar, and re-run your app with yappi.
2) Extract yappi.03.y1.rar, and re-run your app with yappi.

You don't have to run separate tests only below is enough:

import yappi
yappi.start()
Application().run()
yappi.print_stats()

Thanks again,

Original comment by sum...@gmail.com on 22 Feb 2010 at 4:58

Attachments:

• yappi.03.h1.rar
• yappi.03.y1.rar

[GoogleCodeExporter]

And also, could you test the following on your machine, is the test file 
working fine
on your system?

Original comment by sum...@gmail.com on 22 Feb 2010 at 4:59

Attachments:

• test13.py

[GoogleCodeExporter]

Hi,

I tested the distributions you uploaded with extra_compile_args = ["-D 
DEBUG_CALL",
"-D YDEBUG"], the two types of errors I got are attached.

As before, I allways recompiled and reinstalled yappi the following way:

rm -Rf build/
python setup.py install

yappi.03.h1.rar crashed every one testing.
yappi.03.y1.rar crashed most of the time, but one or two times it worked 
without errors.

The test13.py worked right both with yappi.03.h1.rar and yappi.03.y1.rar you
uploaded. The app I'm using is rather complex (a lot of memory allocations, 
recursive
functions, cython-compiled extension modules, etc.) and if there is a related 
bug
somewhere in yappi, this simple test does not triggers it. The app works well 
without
yappi and profiled per-threads previously with python's built-in profile 
module, I
just searched something to measure the hot spots summarized in the whole 
application
in one.

Original comment by mjan...@gmail.com on 22 Feb 2010 at 9:30

Attachments:

• yappi.03.h1_1.txt
• yappi.03.h1_2.txt
• yappi.03.y1_1.txt
• yappi.03.y1_2.txt

[GoogleCodeExporter]

I tried to link _yappi.so with the ccmalloc memory debugger. Output attached.

Original comment by mjan...@gmail.com on 22 Feb 2010 at 10:10

Attachments:

• yappi_ccmalloc.txt

[GoogleCodeExporter]

ccmalloc test was with yappi-0.3beta and

import yappi
yappi.start()
Application().run()
yappi.print_stats()

Original comment by mjan...@gmail.com on 22 Feb 2010 at 10:13

[GoogleCodeExporter]

OK, thank you mjanos. I think I cannot solve this bug without writing a simple
generic memory corruption detector. I will start today, I expect no more than 2 
days
to complete it. However, if it is possible that you can send me the source code 
of
the app you are profiling(if not associated with any license or so), I will 
happily
help you with profiling your application. I am assuming that you cannot send the
source code of the app and going to implement the simple tool. Will inform you 
from here.

Original comment by sum...@gmail.com on 22 Feb 2010 at 11:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Sorry, I can't send the app source...

I ran valgrind the following way:

valgrind --tool=memcheck python ../bin/start.py 2>valgrind.txt

I ran it with your test13.py too, but didn't found error in that.

Original comment by mjan...@gmail.com on 22 Feb 2010 at 11:39

[GoogleCodeExporter]

Ohh, yes I think you found one of the problems. I am assuming this is the 
version
that I sent you, not the original v0.3 right? Can you please run valgrind on 
the orig
v0.3 with this:
import yappi
yappi.start()
Application().run()
yappi.print_stats()

Original comment by sum...@gmail.com on 22 Feb 2010 at 11:49

[GoogleCodeExporter]

Seems that was the version 0.3beta overwritten with _yhashtab.c and _ymem.c. you
uploaded first time.

I ran the test again with a clean 0.3beta copy.

Original comment by mjan...@gmail.com on 23 Feb 2010 at 12:09

Attachments:

• valgrind.txt.gz

[GoogleCodeExporter]

The first errors you send is because of a long function name is written to an 
buffer
which is smaller (off by one). The second one is more tricky, there is a 
problem with
the hash table grow algorithm, I will try to reproduce the problem and send you 
the
emerging version for you to test. Thanks again.

Original comment by sum...@gmail.com on 23 Feb 2010 at 12:21

[GoogleCodeExporter]

Ok, you are welcome.

Original comment by mjan...@gmail.com on 23 Feb 2010 at 12:24

[GoogleCodeExporter]

Hi mjanos. 
Could you confirm this version of yappi works for your app? If possible, 
valgrind
output will be great.

Original comment by sum...@gmail.com on 23 Feb 2010 at 4:31

Attachments:

• yappi-0.4-testing.zip

[GoogleCodeExporter]

Hi,

it doesn't work. It just prints out the following line in infinite loop:

[*] [yappi-err] spush failed, seems hadd failed?

I don't see yappi memory errors in the valgrind output.

Original comment by mjan...@gmail.com on 23 Feb 2010 at 4:58

Attachments:

• valgrind.txt.gz

[GoogleCodeExporter]

Memory errors seems to be fixed right now. Please try this one. Have you tried
printing the stat output?

Original comment by sum...@gmail.com on 23 Feb 2010 at 5:21

Attachments:

• yappi-0.4-testing-2.zip

[GoogleCodeExporter]

It works now. I tested with

import yappi
yappi.start()
Application().run()
yappi.print_stats()

Original comment by mjan...@gmail.com on 23 Feb 2010 at 5:57

[GoogleCodeExporter]

OK. Thanks for all your help Matyas. I will publish 0.4 tonight along with the 
setup
problems. If possible can you make one more test, the same one with the version 
for
some hash table optimization. I am not sure why but I am not able to reproduce 
any of
the errors you faced till now:) 

Original comment by sum...@gmail.com on 23 Feb 2010 at 6:04

Attachments:

• yappi-0.4-testing-3.zip

[GoogleCodeExporter]

Works, tested with:

import yappi
yappi.start()
Application().run()
yappi.stop()

for it in yappi.get_stats(yappi.SORTTYPE_TTOTAL, yappi.SORTORDER_ASCENDING,
yappi.SHOW_ALL):
    print it

Original comment by mjan...@gmail.com on 23 Feb 2010 at 6:14

[GoogleCodeExporter]

Issue #6 is related so closing both of them.

Original comment by sum...@gmail.com on 23 Feb 2010 at 7:11

• Changed state: Verified

[GoogleCodeExporter]

Issue 6 has been merged into this issue.

Original comment by sum...@gmail.com on 23 Feb 2010 at 7:11