benoitc / gaffer

control, watch and launch your applications and jobs over HTTP.
Other
367 stars 34 forks source link

0.5.0 py.test Websocket makes a core dump #33

Closed jul closed 11 years ago

jul commented 11 years ago

So I installed from github because I could not run the tests for 0.4.4 (examples were not working either).

The tests core dump (unless I use strace than I have different outcome)

I hope I can attach file :/

(py27)[gaffer@playground3 gaffer]$ pip list colorama (0.2.5) gaffer (0.5.0) psutil (0.7.1) py (1.4.13) pytest (2.3.5) pyuv (0.10.3) setproctitle (1.1.7) six (1.3.0) tornado (2.4) wsgiref (0.1.2)

at least this is the error message

test/test_user.py:53: test_auth_backend PASSED test/test_user.py:88: test_authenticate PASSED test/test_webchannel.py:45: test_basic_socket Aborted (core dumped)

EDIT result in verbose mode https://gist.github.com/jul/5526637

result + strace interelaced https://gist.github.com/jul/5526623/raw/a28422af3a804390e75d7ec6e38c01c274039149/res%2Bstrace.tx

saghul commented 11 years ago

Can you open the dump file with gdb and post the backtrace?

jul commented 11 years ago

No I can't : I have to use centOS and even by following the howto I can't figure howto enable core dump/or find them.

I reinstalled from source on an empty home dir/user.

I redid a clean install with python setup.py I have of course different results than usual, but by changing pyuv version I experiment the core dump (0.10.3) with 0.8.3 I am stuck in an infinite loop here test/test_controller.py:45: test_basic

saghul commented 11 years ago

IIRC gaffer was adapted to work with pyuv 0.10, so you should use that one. As for the core dumping, doesn't ulimit -c unlimited work for you?

jul commented 11 years ago

It should, I even did a echo /tmp/core.%u.%p > /proc/sys/kernel/core_pattern to find my core dump (because else I don't know where they are), there is nothing... :/

EDIT :: except in my logs

May 6 11:44:20 MTL-WKS-AI514 abrtd: Directory 'ccpp-2013-05-06-11:44:19-817' creation detected May 6 11:44:20 MTL-WKS-AI514 abrt[976]: Saved core dump of pid 817 (/home/gaffer/py27/bin/python2.7) to /var/spool/abrt/ccpp-2013-05-06-11:44:19-817 (81719296 bytes) May 6 11:44:20 MTL-WKS-AI514 abrtd: Executable '/home/gaffer/py27/bin/python2.7' doesn't belong to any package May 6 11:44:20 MTL-WKS-AI514 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-05-06-11:44:19-817' exited with 1 May 6 11:44:20 MTL-WKS-AI514 abrtd: Corrupted or bad directory '/var/spool/abrt/ccpp-2013-05-06-11:44:19-817', deleting May 6 11:45:36 MTL-WKS-AI514 abrtd: Directory 'ccpp-2013-05-06-11:45:35-1556' creation detected May 6 11:45:36 MTL-WKS-AI514 abrt[1749]: Saved core dump of pid 1556 (/home/gaffer/py27/bin/python2.7) to /var/spool/abrt/ccpp-2013-05-06-11:45:35-1556 (67293184 bytes) May 6 11:45:36 MTL-WKS-AI514 abrtd: Executable '/home/gaffer/py27/bin/python2.7' doesn't belong to any package May 6 11:45:36 MTL-WKS-AI514 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-05-06-11:45:35-1556' exited with 1 May 6 11:45:36 MTL-WKS-AI514 abrtd: Corrupted or bad directory '/var/spool/abrt/ccpp-2013-05-06-11:45:35-1556', deleting

EDIT2: This is the strace accept(16, {sa_family=AF_INET, sin_port=htons(36386), sin_addr=inet_addr("127.0.0.1")}, [16]) = 21 fcntl(21, F_GETFL) = 0x2 (flags O_RDWR) fcntl(21, F_SETFL, O_RDWR|O_NONBLOCK) = 0 recvfrom(21, 0x27a7934, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) accept(16, 0x7fff80b054f0, [16]) = -1 EAGAIN (Resource temporarily unavailable) getsockopt(20, SOL_SOCKET, SO_ERROR, [0], [4]) = 0 write(15, "\1\0\0\0\0\0\0\0", 8) = 8 write(2, "python2.7 /home/gaffer/py27/bin/"..., 122) = 122 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f129a11c000 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 tgkill(4016, 4016, SIGABRT) = 0 --- SIGABRT (Aborted) @ 0 (0) --- +++ killed by SIGABRT +++

EDIT 3 : Same problem on 2 differents computer.

jul commented 11 years ago

Okay I know who is the culprit pip install pyuv==0.10.0 solves the problem

[gaffer@MTL-WKS-AI514 gaffer]$ py.test ==================================================================================> ========================== test session starts ============================================================================================================ platform linux2 -- Python 2.7.2 -- pytest-2.3.5 collected 178 items / 2 skipped

test/test_channel.py ... test/test_controller.py ..................... test/test_docopt.py ....................................... test/test_events.py ........ test/test_http.py ............. test/test_keys.py ........................ test/test_lookup.py ........... test/test_manager.py ..................... test/test_message.py ... test/test_pbkdf2.py . test/test_pid_channel.py .... test/test_process.py ........... test/test_sync.py ....... test/test_user.py .... test/test_webchannel.py ...... test/test_webhooks.py . test/test_websocket.py .

==================================================================================> ================ 178 passed, 2 skipped in 29.88 seconds > ===================================================================================================

saghul commented 11 years ago

Well, that's not a solution :-S I haven't seen this problem myself, it would be nice if you could provide a backtrace so I can debug it further in pyuv.

jul commented 11 years ago

okay, I just noticed you are pyuv main dev :) I understand your concern. I will make some efforts, I just am profoundly uncomfortable as a dev with centOS/RH: I have the feeling it does too much things implicitly with vars hidden in cryptic config files for its own good.

jul commented 11 years ago

I forgot about this: http://wiki.python.org/moin/DebuggingWithGdb

jul commented 11 years ago

Arggg, so many years I had a core dump, it was a pain to remember how it works

(py27)[gaffer@playground3 source]$ gdb /usr/local/bin/python2.7 /tmp/core.503.10026 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/bin/python2.7...done.

warning: core file may not match specified executable file. [New Thread 10026] [New Thread 10120] [New Thread 10119] [New Thread 10117] [New Thread 10118] Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/zlib.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/4d/aaf62d9320712a86fff5013a4fdf366c9c3efa Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/time.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/0d/b0b6b9b97439735906f822cabbd649e8647ea4 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/strop.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/50/5088c0334b085bac862fd86e020d012e89ab2b Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/operator.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/ad/b143973b325d9f555dbf2e384fff108c008aad Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/collections.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/f2/d7b38aa463e2d5cfb96501ad3a731092c4389a Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/itertools.so Try: yum --disablerepo='' --enablerepo='-debug_' install /usr/lib/debug/.build-id/67/e7092cf506b854ae2c3a6b968386599fc14cf4 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/bisect.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/e3/7a52e22b185bfa1d3f68004cb72e203a097e00 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/heapq.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/8a/571ed2cb4f88ae2fbbf027c00c7855399cdc7a Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/functools.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/d0/86633687d661aa8908fcd522807ac48e29d9d0 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/locale.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/41/63f0cad71dc965c501a546010697f233c0caf6 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/struct.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/34/c6d0f3305be63c3fd8d2c45b883e88748def47 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/math.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/16/0409010a55d18ee76835e61b4e36f758a889ec Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/binascii.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/df/ff53309a2d7a877e708306122afceec5473d8a Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/hashlib.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/b1/e2155d4a90155c66dad664a46e0df630e501ac Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/random.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/93/f8da82fd1cf92a4f473fccfe17f844b7ec6f63 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/cStringIO.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/46/21fc0db83d921ce99d5192ff6ede439e63b446 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/fcntl.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/96/6bbe13f988a9d40461517c3f671e33a53f5438 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/io.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/c9/18ff72dd295c5b9f40dc535df6f0d2fdfc7381 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/array.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/2c/4c15e2ac28525de2a8dfa1400b81cec16dab5b Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/parser.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/b6/c08d9f22f3a39659648b641b31d833ef5db9b0 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/termios.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/d6/ced659b8df5428f5c321b7d5771e0707f4218f Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/site-packages/pyuv.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/67/356957678f94a1182acd8bb858b8cbcbf4949b Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/json.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/1b/205c1d1bbe82b3b02fb51da661c73695201e69 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/site-packages/gaffer-0.5.0-py2.7-linux-x8664.egg/gaffer/sync.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/41/6bf413d8a0aba196feabd93eb5d128ef880b36 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/socket.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/e3/a3c0ff80b4f9871b457d631c73b2b87a6f9a76 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/ssl.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/5d/5a36a6cd3147368b45048be09e34ee784b4418 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/grp.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/c7/afd2e607864121b4d2b46451d30a076c616cda Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/resource.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/d2/3edf8b41f7786186f4b5940c7f80a64d7f5f80 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/datetime.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/4d/be28070b4bff0f1f7ec3315c90dd54e33120c3 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/select.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/5a/3605594225b5c2d9e74bbc374b67286777080e Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/multiprocessing.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/97/f62eb86ce5cf07cc727ae7a15dac0b8381b2f9 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/cPickle.so Try: yum --disablerepo='' --enablerepo='-debug_' install /usr/lib/debug/.build-id/51/95161275f5ea044a61de1bf98ebd0db061f0ef Missing separate debuginfo for /home/gaffer/.python-eggs/setproctitle-1.1.7-py2.7-linux-x8664.egg-tmp/setproctitle.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/8c/3d8de2cb8ea8dcbfa4d1a9c2804a626aff501c Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/ctypes.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/b2/927c149d20de6b717980d4a186528fa892b3c6 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/csv.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/2e/291a17f9df13c2fdea9f582db6253b180e96df Missing separate debuginfo for /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutilposix.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/9b/e777149c9581e15e7343a73b5a573dc2e5d036 Missing separate debuginfo for /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutillinux.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/11/0b9a2d859ca4363774c05761a288c67b23b4c0 Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/sqlite3.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/82/ab4400534dfd1ef15891528ed2d5db861a53fa Missing separate debuginfo for /home/gaffer/source/py27/lib/python2.7/lib-dynload/unicodedata.so Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/fa/e474f1294ab66cf514cb4648e0bda38693b3de Missing separate debuginfo for Try: yum --disablerepo='' --enablerepo='-debug' install /usr/lib/debug/.build-id/fd/395df091337c1b475d3a2b95fd65ca28310293 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/zlib.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/zlib.so Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/time.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/time.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/strop.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/strop.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/operator.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/operator.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_collections.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_collections.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/itertools.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/itertools.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_bisect.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_bisect.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_heapq.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_heapq.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_functools.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_functools.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_locale.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_locale.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_struct.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_struct.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/math.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/math.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/binascii.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/binascii.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_hashlib.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_hashlib.so Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_random.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_random.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/cStringIO.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/cStringIO.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/fcntl.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/fcntl.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_io.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_io.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/array.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/array.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/parser.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/parser.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/termios.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/termios.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/site-packages/pyuv.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/site-packages/pyuv.so Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_json.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_json.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/site-packages/gaffer-0.5.0-py2.7-linux-x86_64.egg/gaffer/sync.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/site-packages/gaffer-0.5.0-py2.7-linux-x86_64.egg/gaffer/sync.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_socket.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_socket.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_ssl.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_ssl.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/grp.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/grp.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/resource.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/resource.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/datetime.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/datetime.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/select.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/select.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_multiprocessing.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_multiprocessing.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/cPickle.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/cPickle.so Reading symbols from /home/gaffer/.python-eggs/setproctitle-1.1.7-py2.7-linux-x86_64.egg-tmp/setproctitle.so...done. Loaded symbols for /home/gaffer/.python-eggs/setproctitle-1.1.7-py2.7-linux-x86_64.egg-tmp/setproctitle.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_ctypes.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_ctypes.so Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_csv.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_csv.so Reading symbols from /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutil_posix.so...done. Loaded symbols for /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutil_posix.so Reading symbols from /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutil_linux.so...done. Loaded symbols for /home/gaffer/.python-eggs/psutil-0.7.1-py2.7-linux-x86_64.egg-tmp/_psutil_linux.so Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/_sqlite3.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/_sqlite3.so Reading symbols from /usr/lib64/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsqlite3.so.0 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /home/gaffer/source/py27/lib/python2.7/lib-dynload/unicodedata.so...done. Loaded symbols for /home/gaffer/source/py27/lib/python2.7/lib-dynload/unicodedata.so Core was generated by `/home/gaffer/source/py27/bin/python2.7 /home/gaffer/source/py27/bin/py.test tes'. Program terminated with signal 6, Aborted.

0 0x00000035798328a5 in raise () from /lib64/libc.so.6

Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6_4.1.x86_64 libcom_err-1.41.12-14.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libuuid-2.17.2-12.9.el6.x86_64 openssl-1.0.0-27.el6_4.2.x86_64 sqlite-3.6.20-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt

0 0x00000035798328a5 in raise () from /lib64/libc.so.6

1 0x0000003579834085 in abort () from /lib64/libc.so.6

2 0x000000357982ba1e in __assert_fail_base () from /lib64/libc.so.6

3 0x000000357982bae0 in __assert_fail () from /lib64/libc.so.6

4 0x00007f4af993d4d6 in uv__io_stop (loop=0x7f4af9b87f00, w=0x356ea68, events=1) at src/unix/core.c:688

5 0x00007f4af9947f5f in uv_read_stop (stream=0x356e9f8) at src/unix/stream.c:1286

6 0x00007f4af9948021 in uv__stream_close (handle=0x356e9f8) at src/unix/stream.c:1352

7 0x00007f4af9942036 in uv__pipe_close (handle=0x356e9f8) at src/unix/pipe.c:153

8 0x00007f4af993c4d0 in uv_close (handle=0x356e9f8, close_cb=0x7f4af9927ef0 ) at src/unix/core.c:91

9 0x00007f4af9930421 in Handle_tp_dealloc (self=0x356e9b0) at src/handle.c:214

10 0x000000000050144e in cell_dealloc (op=0x33387c0) at Objects/cellobject.c:50

11 0x00000000004674dd in tupledealloc (op=0x3452fc8) at Objects/tupleobject.c:220

12 0x000000000050b2a7 in func_dealloc (op=0x2a98500) at Objects/funcobject.c:461

13 0x00007f4af9926a31 in Timer_tp_clear (self=0x33493f0) at src/timer.c:207

14 0x00007f4af99303f4 in Handle_tp_dealloc (self=0x33493f0) at src/handle.c:235

15 0x00007f4af9927f27 in on_handle_dealloc_close (handle=0x3349430) at src/handle.c:86

16 0x00007f4af993c80e in uv__finish_close (handle=0x3349430) at src/unix/core.c:202

17 0x00007f4af993c859 in uv__run_closing_handles (loop=0x7f4af9b87f00) at src/unix/core.c:216

18 0x00007f4af993cafd in uv_run (loop=0x7f4af9b87f00, mode=UV_RUN_DEFAULT) at src/unix/core.c:314

19 0x00007f4af993b9ef in Loop_func_run (self=0x333b9f0, args=) at src/loop.c:92

20 0x00000000004a74f0 in call_function (f=, throwflag=) at Python/ceval.c:4021

21 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

22 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

23 call_function (f=, throwflag=) at Python/ceval.c:4042

24 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

25 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x3330130, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x7f4b028c6068, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

26 0x000000000050b148 in function_call (func=0x332c758, arg=0x7f4b028c6050, kw=0x3599570) at Objects/funcobject.c:526

27 0x0000000000419507 in PyObject_Call (func=0x332c758, arg=, kw=) at Objects/abstract.c:2529

28 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

29 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

30 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb07a3b0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x333b7a0, kwcount=2, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

31 0x000000000050b148 in function_call (func=0x7f4afb09d9b0, arg=0x7f4b028c6050, kw=0x2639380) at Objects/funcobject.c:526

32 0x0000000000419507 in PyObject_Call (func=0x7f4afb09d9b0, arg=, kw=) at Objects/abstract.c:2529

33 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

34 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

35 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

36 call_function (f=, throwflag=) at Python/ceval.c:4042

37 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

38 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

39 call_function (f=, throwflag=) at Python/ceval.c:4042

40 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

41 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d5a30, globals=, locals=,

args=<value optimized out>, argcount=2, kws=0x3452d10, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

---Type to continue, or q to quit---

42 0x000000000050b148 in function_call (func=0x7f4afc29e6e0, arg=0x34522d8, kw=0x3188210) at Objects/funcobject.c:526

43 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e6e0, arg=, kw=) at Objects/abstract.c:2529

44 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

45 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

46 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb732930, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x267b078, kwcount=1, defs=0x0, defcount=0, closure=0x34513f8) at Python/ceval.c:3253

47 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

48 call_function (f=, throwflag=) at Python/ceval.c:4042

49 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

50 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

51 call_function (f=, throwflag=) at Python/ceval.c:4042

52 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

53 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f5b0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x34524a0, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

54 0x000000000050b148 in function_call (func=0x7f4afb06ede8, arg=0x7f4b028c6050, kw=0x3190c10) at Objects/funcobject.c:526

55 0x0000000000419507 in PyObject_Call (func=0x7f4afb06ede8, arg=, kw=) at Objects/abstract.c:2529

56 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

57 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

58 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

59 call_function (f=, throwflag=) at Python/ceval.c:4042

60 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

61 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

62 call_function (f=, throwflag=) at Python/ceval.c:4042

63 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

64 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d5a30, globals=, locals=,

args=<value optimized out>, argcount=2, kws=0x3470848, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

65 0x000000000050b148 in function_call (func=0x7f4afc29e6e0, arg=0x34526c8, kw=0x3608c10) at Objects/funcobject.c:526

66 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e6e0, arg=, kw=) at Objects/abstract.c:2529

67 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

68 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

69 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb732930, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x3466a88, kwcount=1, defs=0x0, defcount=0, closure=0x3454a28) at Python/ceval.c:3253

70 0x000000000050b148 in function_call (func=0x2f7dd70, arg=0x7f4b028c6050, kw=0x30c8900) at Objects/funcobject.c:526

71 0x0000000000419507 in PyObject_Call (func=0x2f7dd70, arg=, kw=) at Objects/abstract.c:2529

72 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

73 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

74 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f7b0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x35ef908, kwcount=0, defs=0x0, defcount=0, closure=0x31f6a00) at Python/ceval.c:3253

75 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

76 call_function (f=, throwflag=) at Python/ceval.c:4042

77 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

78 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f8b0, globals=, locals=,

args=<value optimized out>, argcount=2, kws=0x3451f50, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

79 0x000000000050b148 in function_call (func=0x7f4afb07b140, arg=0x346b098, kw=0x33df670) at Objects/funcobject.c:526

80 0x0000000000419507 in PyObject_Call (func=0x7f4afb07b140, arg=, kw=) at Objects/abstract.c:2529

---Type to continue, or q to quit---

81 0x00000000004220df in instancemethod_call (func=0x7f4afb07b140, arg=0x346b098, kw=0x33df670) at Objects/classobject.c:2578

82 0x0000000000419507 in PyObject_Call (func=0x2f79230, arg=, kw=) at Objects/abstract.c:2529

83 0x00000000004a0dc3 in PyEval_CallObjectWithKeywords (func=0x2f79230, arg=0x2f7a890, kw=) at Python/ceval.c:3890

84 0x0000000000424376 in PyInstance_New (klass=, arg=0x2f7a890, kw=0x33df670) at Objects/classobject.c:571

85 0x0000000000419507 in PyObject_Call (func=0x7f4afb0dcbb0, arg=, kw=) at Objects/abstract.c:2529

86 0x00000000004a6a60 in do_call (f=, throwflag=) at Python/ceval.c:4239

87 call_function (f=, throwflag=) at Python/ceval.c:4044

88 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

89 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f830, globals=, locals=,

args=<value optimized out>, argcount=2, kws=0x7f4b028c6068, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

90 0x000000000050b148 in function_call (func=0x7f4afb07b050, arg=0x3452f38, kw=0x306e110) at Objects/funcobject.c:526

91 0x0000000000419507 in PyObject_Call (func=0x7f4afb07b050, arg=, kw=) at Objects/abstract.c:2529

92 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

93 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

94 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f730, globals=, locals=,

args=<value optimized out>, argcount=3, kws=0x2dee820, kwcount=0, defs=0x7f4afb0773a8, defcount=1, closure=0x0) at Python/ceval.c:3253

95 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

96 call_function (f=, throwflag=) at Python/ceval.c:4042

97 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

98 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f4b0, globals=, locals=,

args=<value optimized out>, argcount=1, kws=0x3360fc8, kwcount=1, defs=0x7f4afb0edda0, defcount=2, closure=0x0) at Python/ceval.c:3253

99 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

100 call_function (f=, throwflag=) at Python/ceval.c:4042

101 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

102 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb06f3b0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x28a4640, kwcount=2, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

103 0x000000000050b148 in function_call (func=0x7f4afb06ec08, arg=0x7f4b028c6050, kw=0x2f008d0) at Objects/funcobject.c:526

104 0x0000000000419507 in PyObject_Call (func=0x7f4afb06ec08, arg=, kw=) at Objects/abstract.c:2529

105 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

106 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

107 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

108 call_function (f=, throwflag=) at Python/ceval.c:4042

109 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

110 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

111 call_function (f=, throwflag=) at Python/ceval.c:4042

112 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

113 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d59b0, globals=, locals=,

args=<value optimized out>, argcount=1, kws=0x2882fe0, kwcount=2, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

114 0x000000000050b148 in function_call (func=0x7f4afc29e668, arg=0x31bdc50, kw=0x3071a20) at Objects/funcobject.c:526

115 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e668, arg=, kw=) at Objects/abstract.c:2529

116 0x00000000004220df in instancemethod_call (func=0x7f4afc29e668, arg=0x31bdc50, kw=0x3071a20) at Objects/classobject.c:2578

117 0x0000000000419507 in PyObject_Call (func=0x303af50, arg=, kw=) at Objects/abstract.c:2529

118 0x00000000004213dd in instance_call (func=0x7f4afc27d758, arg=0x7f4b028c6050, kw=0x3071a20) at Objects/classobject.c:2129

119 0x0000000000419507 in PyObject_Call (func=0x7f4afc27d758, arg=, kw=) at Objects/abstract.c:2529

120 0x00000000004a6a60 in do_call (f=, throwflag=) at Python/ceval.c:4239

---Type to continue, or q to quit---

121 call_function (f=, throwflag=) at Python/ceval.c:4044

122 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

123 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb7327b0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x2893218, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

124 0x000000000050b148 in function_call (func=0x7f4afb0e1050, arg=0x7f4b028c6050, kw=0x293f8d0) at Objects/funcobject.c:526

125 0x0000000000419507 in PyObject_Call (func=0x7f4afb0e1050, arg=, kw=) at Objects/abstract.c:2529

126 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

127 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

128 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

129 call_function (f=, throwflag=) at Python/ceval.c:4042

130 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

131 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

132 call_function (f=, throwflag=) at Python/ceval.c:4042

133 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

134 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d59b0, globals=, locals=,

args=<value optimized out>, argcount=1, kws=0x7f4afc270e30, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

135 0x000000000050b148 in function_call (func=0x7f4afc29e668, arg=0x7f4afa1cde90, kw=0x27a4740) at Objects/funcobject.c:526

136 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e668, arg=, kw=) at Objects/abstract.c:2529

137 0x00000000004220df in instancemethod_call (func=0x7f4afc29e668, arg=0x7f4afa1cde90, kw=0x27a4740) at Objects/classobject.c:2578

138 0x0000000000419507 in PyObject_Call (func=0x7f4afa1c8c30, arg=, kw=) at Objects/abstract.c:2529

139 0x00000000004213dd in instance_call (func=0x7f4afc27db90, arg=0x7f4b028c6050, kw=0x27a4740) at Objects/classobject.c:2129

140 0x0000000000419507 in PyObject_Call (func=0x7f4afc27db90, arg=, kw=) at Objects/abstract.c:2529

141 0x00000000004a6a60 in do_call (f=, throwflag=) at Python/ceval.c:4239

142 call_function (f=, throwflag=) at Python/ceval.c:4044

143 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

144 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

145 call_function (f=, throwflag=) at Python/ceval.c:4042

146 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

147 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

148 call_function (f=, throwflag=) at Python/ceval.c:4042

149 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

150 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afb732330, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x7f4afa1c3c80, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

151 0x000000000050b148 in function_call (func=0x7f4afb0dfe60, arg=0x7f4b028c6050, kw=0x279fd00) at Objects/funcobject.c:526

152 0x0000000000419507 in PyObject_Call (func=0x7f4afb0dfe60, arg=, kw=) at Objects/abstract.c:2529

153 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

154 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

155 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

156 call_function (f=, throwflag=) at Python/ceval.c:4042

157 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

158 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

159 call_function (f=, throwflag=) at Python/ceval.c:4042

160 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

161 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d59b0, globals=, locals=,

args=<value optimized out>, argcount=1, kws=0x7f4afa1c3728, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

---Type to continue, or q to quit---

162 0x000000000050b148 in function_call (func=0x7f4afc29e668, arg=0x7f4b02885f10, kw=0x2524390) at Objects/funcobject.c:526

163 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e668, arg=, kw=) at Objects/abstract.c:2529

164 0x00000000004220df in instancemethod_call (func=0x7f4afc29e668, arg=0x7f4b02885f10, kw=0x2524390) at Objects/classobject.c:2578

165 0x0000000000419507 in PyObject_Call (func=0x7f4afc2c85a0, arg=, kw=) at Objects/abstract.c:2529

166 0x00000000004213dd in instance_call (func=0x7f4afc27de60, arg=0x7f4b028c6050, kw=0x2524390) at Objects/classobject.c:2129

167 0x0000000000419507 in PyObject_Call (func=0x7f4afc27de60, arg=, kw=) at Objects/abstract.c:2529

168 0x00000000004a6a60 in do_call (f=, throwflag=) at Python/ceval.c:4239

169 call_function (f=, throwflag=) at Python/ceval.c:4044

170 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

171 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d5cb0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x2461218, kwcount=0, defs=0x7f4afc271920, defcount=2, closure=0x0) at Python/ceval.c:3253

172 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

173 call_function (f=, throwflag=) at Python/ceval.c:4042

174 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

175 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4b02897a30, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

176 0x00000000004a90e2 in PyEval_EvalCode (co=, globals=, locals=)

at Python/ceval.c:667

177 0x00000000004c959e in run_mod (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/",

start=<value optimized out>, globals=0x23f8160, locals=0x23f8160, closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:1353

178 PyRun_FileExFlags (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/", start=,

globals=0x23f8160, locals=0x23f8160, closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:1339

179 0x00000000004c97b4 in PyRun_SimpleFileExFlags (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/",

closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:943

180 0x0000000000414bd2 in Py_Main (argc=, argv=) at Modules/main.c:639

181 0x000000357981ecdd in __libc_start_main () from /lib64/libc.so.6

182 0x0000000000413cc9 in _start ()

(gdb)

jul commented 11 years ago

.... CONT ...

153 0x00000000004a5e3a in ext_do_call (f=, throwflag=) at Python/ceval.c:4334

154 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2705

155 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

156 call_function (f=, throwflag=) at Python/ceval.c:4042

157 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

158 0x00000000004a8465 in fast_function (f=, throwflag=) at Python/ceval.c:4107

159 call_function (f=, throwflag=) at Python/ceval.c:4042

160 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

161 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d59b0, globals=, locals=,

args=<value optimized out>, argcount=1, kws=0x7f4afa1c3728, kwcount=1, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

---Type to continue, or q to quit---

162 0x000000000050b148 in function_call (func=0x7f4afc29e668, arg=0x7f4b02885f10, kw=0x2524390) at Objects/funcobject.c:526

163 0x0000000000419507 in PyObject_Call (func=0x7f4afc29e668, arg=, kw=) at Objects/abstract.c:2529

164 0x00000000004220df in instancemethod_call (func=0x7f4afc29e668, arg=0x7f4b02885f10, kw=0x2524390) at Objects/classobject.c:2578

165 0x0000000000419507 in PyObject_Call (func=0x7f4afc2c85a0, arg=, kw=) at Objects/abstract.c:2529

166 0x00000000004213dd in instance_call (func=0x7f4afc27de60, arg=0x7f4b028c6050, kw=0x2524390) at Objects/classobject.c:2129

167 0x0000000000419507 in PyObject_Call (func=0x7f4afc27de60, arg=, kw=) at Objects/abstract.c:2529

168 0x00000000004a6a60 in do_call (f=, throwflag=) at Python/ceval.c:4239

169 call_function (f=, throwflag=) at Python/ceval.c:4044

170 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

171 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4afc2d5cb0, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x2461218, kwcount=0, defs=0x7f4afc271920, defcount=2, closure=0x0) at Python/ceval.c:3253

172 0x00000000004a7184 in fast_function (f=, throwflag=) at Python/ceval.c:4117

173 call_function (f=, throwflag=) at Python/ceval.c:4042

174 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2666

175 0x00000000004a8fd7 in PyEval_EvalCodeEx (co=0x7f4b02897a30, globals=, locals=,

args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3253

176 0x00000000004a90e2 in PyEval_EvalCode (co=, globals=, locals=)

at Python/ceval.c:667

177 0x00000000004c959e in run_mod (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/",

start=<value optimized out>, globals=0x23f8160, locals=0x23f8160, closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:1353

178 PyRun_FileExFlags (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/", start=,

globals=0x23f8160, locals=0x23f8160, closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:1339

179 0x00000000004c97b4 in PyRun_SimpleFileExFlags (fp=0x2453180, filename=0x7fff8d6df73b "/home/gaffer/source/py27/bin/py.test test/",

closeit=1, flags=0x7fff8d6deaa0) at Python/pythonrun.c:943

180 0x0000000000414bd2 in Py_Main (argc=, argv=) at Modules/main.c:639

181 0x000000357981ecdd in __libc_start_main () from /lib64/libc.so.6

182 0x0000000000413cc9 in _start ()

saghul commented 11 years ago

According to the backtrace you are hitting this issue: https://github.com/saghul/pyuv/issues/54 which is not fixable unfortunately. Looks like 2 Poll handles are being used to monitor the same fd, and that makes libuv assert. Maybe @benoitc has some insight on where in gaffer's code that may happen.

jul commented 11 years ago

What I don't understand is how

python setup.py install

Called from the github clone of current code did not followed the requirements.txt where pyuv==0.10.0 was explicitly set.

This version requirements is at my opinion explicitly made so we don't hit that bug. Maybe I should also debug my installation ?

saghul commented 11 years ago

Because distutils / setuptools don't follow requirements.txt, it's pip who does.

jul commented 11 years ago

Saghul : would this other commit be safer too ? I just force on setup.py pyuv == 0.10.0 as long as the sigsev bug is not fixed. https://github.com/jul/gaffer/commit/b2f65585346dc1989adae7dd79b178c00a9546aa

saghul commented 11 years ago

@jul Not really. The problem is still there but there is no assert on 0.10.0 apparently. This should be fixed in gaffer, since libuv (and hence pyuv) cannot work with more than one Poll handle for the same fd.

benoitc commented 11 years ago

not sure where it happe actually. I suspect there is something with tornado_pyuv and its use internally but I didn't find anything yet

saghul commented 11 years ago

Could be, yes. Do we use Poll handles by hand anywhere in gaffer?

jul commented 11 years ago

@saghul : anyway when I run the test 3 times in a row with pyuv==0.10.0 I have results that varies. It stinks of an heisenbug due to an inconsistent internal state. Maybe a critical section is not locked?

https://gist.github.com/jul/5541067

saghul commented 11 years ago

@jul As I told you, the problem was already there on version 0.10.0, but there was no assert to trigger the failure, so results are unexpected. This will not be fixed in libuv so it's up to applications to make sure they don't create more than one poll handle for the same fd. Unfortunately adding error checking for this is not trivial inside libuv, that's why there is an assert there. One way to debug this would be to subclass Poll and have a WeakKeyDisctionary mapping poll handle instances to fds. Then in MyPoll.init you could check if the fd is in the values of that dict and print a backtrace.

jul commented 11 years ago

@saghul : I have a short deadline on my project .... I guess I wont be able to help much more. Your problem kind of looks like a singleton: if someone wants to create a new poll handle for a fd, while one already exists, then maybe you could return the already existing one. The problem is I am not fluent enough in python C extension to know how to reproduce the borg idiom (using a global or a class variable) and I had a hard time reading/understanding your C code (I have not coded a lot in C these last years). I won't be helpful.

saghul commented 11 years ago

I understand, no worries. The problem is that avoiding the issue comes at a performance cost, also, I didn't have the time to try to implement it myself. I would be helpful if you can write a short test script which triggers the issue, in order to avoid possible side effects between tests.

jul commented 11 years ago

@saghul : It is a state bug I don't know how to trigger these bugs in a deterministic fashion. I guess you might want a library for stack/pointer debugging http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging

saghul commented 11 years ago

I know how to trigger it, it's described in the issue I linked earlier :-) What I don't know at this point is where the criteria is met in gaffer.

jul commented 11 years ago

Seems weired: in IOLoop.add_handler you already raised an exception if you try to repoll an fd already polled. But I don't understand why self._handlers is an instance variable, I think you really should make it a class variable. The problem with your exception is that if two distinct instances try to add an handler on the same fd you won't catch it.

benoitc commented 11 years ago

@jul there maybe a problem due to the fact that it is creating different IOPool instances around. instead of raising here, we should probably try to manage a global IOPool.

This kind of problem is solved in coming version since it's not using anymore tornado but, maybe a quick patch would do the trick there.

benoitc commented 11 years ago

fixed in bf1af98bda0e0661ffa298adc51269fa8d4d1866