search

python / asyncio

asyncio historical repository
https://docs.python.org/3/library/asyncio.html
1.04k stars 177 forks source link

_lock attribute errors #443

Closed frensjan closed 7 years ago

frensjan commented 7 years ago

I'm witnessing quite peculiar errors when using asyncio.locks.Lock in a with construct, for example:

Traceback (most recent call last):
  << non asyncio traceback omitted >>
  File "/usr/lib64/python3.4/asyncio/locks.py", line 37, in __exit__
    self._lock.release()
AttributeError: '_ContextManager' object has no attribute '_lock'

And also:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/asyncio/locks.py", line 37, in __exit__
    self._lock.release()
AttributeError: 'tuple' object has no attribute '_lock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  << non asyncio traceback omitted >>
  File "/usr/lib64/python3.4/asyncio/locks.py", line 39, in __exit__
    self._lock = None  # Crudely prevent reuse.
AttributeError: 'tuple' object has no attribute '_lock'

The code behind these tracebacks boils done to something simple as:

with (yield from lock):
    len_buffer = yield from self.readexactly(8)
    frame_len = struct.unpack('Q', len_buffer)[0]
    frame = yield from self.readexactly(frame_len)

The lock serializes multiple readers on the same connection. I'm using a lock because the asynchronous code is interacting with threaded code.

I'm reeeally confused as the code on line 37 and 39 of asyncio.locks looks up _lock on _ContextManager:

     12 class _ContextManager:
     13     """Context manager.
     14 
     15     This enables the following idiom for acquiring and releasing a
     16     lock around a block:
     17 
     18         with (yield from lock):
     19             <block>
     20 
     21     while failing loudly when accidentally using:
     22 
     23         with lock:
     24             <block>
     25     """
     26 
     27     def __init__(self, lock):
     28         self._lock = lock
     29 
     30     def __enter__(self):
     31         # We have no use for the "as ..."  clause in the with
     32         # statement for locks.
     33         return None
     34 
     35     def __exit__(self, *args):
     36         try:
     37             self._lock.release()
     38         finally:
     39             self._lock = None  # Crudely prevent reuse.

As far as I can see, it always has a _lock member, unless some other code would delete it. Even more curious is the second AttributeError message: _'tuple' object has no attribute 'lock'.

Any help / suggestions on where to poke to get to a root cause would be very much appreciated.

I'm running python 3.4.3 on Centos Linux 7 with kernel 3.10.0-327.36.1.el7.x86_64.

asvetlov commented 7 years ago

asyncio locks are not intended to be thread-safe. They are event-loop safe only. Without looking on full source I have no idea how do you interact between asyncio loop and threaded code but tracebacks are very suspicious.

frensjan commented 7 years ago

Suspicious indeed!

I'm interacting between the threaded and asyncio parts through loop.call_soon_threadsafe(...). This is the proper way to move between threads outside of the event loop right?

asvetlov commented 7 years ago

Yes, loop.call_soon_threadsafe(...) is intended for making loop code calls from threads. janus provides a communication channels with queue interface.

gvanrossum commented 7 years ago

Probably just best to close this issue.

frensjan commented 7 years ago

Happy to close the issue, but would be nice to solve it or at least have a clue on where to look.

frensjan commented 7 years ago

I'm fighting segmentation faults at the same time ... so there may be something else brewing here. 

AttributeError: 'tuple' object has no attribute '_lock'

should be impossible given it's stack trace.

The segmentation faults occur in development on Fedora 24 with python 3.5.1:

# gdb python3.5 ccpp-2016-10-11-12:40:41-18010/coredump
GNU gdb (GDB) Fedora 7.11.1-75.fc24
...
Reading symbols from python3.5...Reading symbols from /usr/lib/debug/usr/bin/python3.5.debug...done.
done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/frens-jan/Workspaces/tgho/bndl/bndl/testvenv/bin/python -m bndl.util.supe'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f8b0ac29471 in subtype_dealloc (self=0x7f8afc0466d8)
    at /usr/src/debug/Python-3.5.1/Objects/typeobject.c:1182
1182            _PyObject_GC_UNTRACK(self);
[Current thread is 1 (Thread 0x7f8b0b22f700 (LWP 18010))]
Missing separate debuginfos, use: dnf debuginfo-install krb5-libs-1.14.3-8.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
(gdb) bt
#0  0x00007f8b0ac29471 in subtype_dealloc (self=<Task at remote 0x7f8afc0466d8>)
    at /usr/src/debug/Python-3.5.1/Objects/typeobject.c:1182
#1  0x00007f8b0abe8947 in method_dealloc (im=0x7f8afc883e08) at /usr/src/debug/Python-3.5.1/Objects/classobject.c:198
#2  0x00007f8b0ac285a9 in clear_slots (type=type@entry=0x560219f0fa88, 
    self=self@entry=<Handle at remote 0x7f8afc035948>) at /usr/src/debug/Python-3.5.1/Objects/typeobject.c:1044
#3  0x00007f8b0ac29506 in subtype_dealloc (self=<Handle at remote 0x7f8afc035948>)
    at /usr/src/debug/Python-3.5.1/Objects/typeobject.c:1200
#4  0x00007f8b0ac8caad in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x56021a01ff08, for file /usr/lib64/python3.5/asyncio/base_events.py, line 1239, in _run_once (self=<_UnixSelectorEventLoop(_coroutine_wrapper_set=False, _current_handle=None, _ready=<collections.deque at remote 0x7f8afd39a250>, _closed=False, _task_factory=None, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f8afc868748>, _epoll=<select.epoll at remote 0x7f8b0b1b8e58>, _fd_to_key={4: <SelectorKey at remote 0x7f8afcac8a98>, 6: <SelectorKey at remote 0x7f8afcac8e08>, 7: <SelectorKey at remote 0x7f8afcac8e60>, 8: <SelectorKey at remote 0x7f8afc873048>, 9: <SelectorKey at remote 0x7f8afc873830>, 10: <SelectorKey at remote 0x7f8afc873af0>, 11: <SelectorKey at remote 0x7f8afc87b620>, 12: <SelectorKey at remote 0x7f8afc87b7d8>, 13: <SelectorKey at remote 0x7f8afc889af0>, 14: <SelectorKey at remote 0x7f8afc884678>, 15: <SelectorKey at remote 0x7f8afc025eb8>, 16: <SelectorKey at remote 0x7f8afc889db0>, 17: <SelectorKey at remote 0x7f8afc01a258>, 18: <SelectorKey at remote 0x7f8afc...(truncated), 
    throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:1414
#5  0x00007f8b0ac8fd3b in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7ffdaa02ef50, 
    func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4783
#6  call_function (oparg=<optimized out>, pp_stack=0x7ffdaa02ef50) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
#7  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f8afe4b7240, for file /usr/lib64/python3.5/asyncio/base_events.py, line 295, in run_forever (self=<_UnixSelectorEventLoop(_coroutine_wrapper_set=False, _current_handle=None, _ready=<collections.deque at remote 0x7f8afd39a250>, _closed=False, _task_factory=None, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f8afc868748>, _epoll=<select.epoll at remote 0x7f8b0b1b8e58>, _fd_to_key={4: <SelectorKey at remote 0x7f8afcac8a98>, 6: <SelectorKey at remote 0x7f8afcac8e08>, 7: <SelectorKey at remote 0x7f8afcac8e60>, 8: <SelectorKey at remote 0x7f8afc873048>, 9: <SelectorKey at remote 0x7f8afc873830>, 10: <SelectorKey at remote 0x7f8afc873af0>, 11: <SelectorKey at remote 0x7f8afc87b620>, 12: <SelectorKey at remote 0x7f8afc87b7d8>, 13: <SelectorKey at remote 0x7f8afc889af0>, 14: <SelectorKey at remote 0x7f8afc884678>, 15: <SelectorKey at remote 0x7f8afc025eb8>, 16: <SelectorKey at remote 0x7f8afc889db0>, 17: <SelectorKey at remote 0x7f8afc01a258>, 18: <SelectorKey at remote 0x7f8af...(truncated), 
    throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#8  0x00007f8b0ac8fd3b in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7ffdaa02f090, 
    func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4783
#9  call_function (oparg=<optimized out>, pp_stack=0x7ffdaa02f090) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
#10 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x56021a01f868, for file /home/frens-jan/Workspaces/tgho/bndl/bndl/bndl/net/run.py, line 42, in run_nodes (started_signal=None, stop_signal=None, nodes=(<Worker(servers={'tcp://localhost.localdomain:5006': <Server(_loop=<_UnixSelectorEventLoop(_coroutine_wrapper_set=False, _current_handle=None, _ready=<collections.deque at remote 0x7f8afd39a250>, _closed=False, _task_factory=None, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f8afc868748>, _epoll=<select.epoll at remote 0x7f8b0b1b8e58>, _fd_to_key={4: <SelectorKey at remote 0x7f8afcac8a98>, 6: <SelectorKey at remote 0x7f8afcac8e08>, 7: <SelectorKey at remote 0x7f8afcac8e60>, 8: <SelectorKey at remote 0x7f8afc873048>, 9: <SelectorKey at remote 0x7f8afc873830>, 10: <SelectorKey at remote 0x7f8afc873af0>, 11: <SelectorKey at remote 0x7f8afc87b620>, 12: <SelectorKey at remote 0x7f8afc87b7d8>, 13: <SelectorKey at remote 0x7f8afc889af0>, 14: <SelectorKey at remote 0x7f8afc884678>, 15: <SelectorKey at remote 0x7f8afc025eb8>, 16:...(truncated), 
    throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#11 0x00007f8b0ac91753 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, 
    locals=locals@entry=0x0, args=args@entry=0x7f8b031c5b70, argcount=1, kws=0x7f8b031c5b78, kwcount=0, defs=0x0, 
    defcount=0, kwdefs={'started_signal': None, 'stop_signal': None}, closure=0x0, name='run_nodes', 
    qualname='run_nodes') at /usr/src/debug/Python-3.5.1/Python/ceval.c:3995
#12 0x00007f8b0ac8de19 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, 
    pp_stack=0x7ffdaa02f2a0, func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4793
#13 call_function (oparg=<optimized out>, pp_stack=0x7ffdaa02f2a0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
#14 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f8b031c59d0, for file /home/frens-jan/Workspaces/tgho/bndl/bndl/bndl/compute/worker.py, line 40, in main (conf=<Config(values={'bndl.compute.worker_count': '24'}) at remote 0x7f8b02702d68>, args=<Namespace(listen_addresses=['tcp://localhost.localdomain:5000'], seeds=['tcp://localhost.localdomain:5000']) at remote 0x7f8afc8684a8>, listen_addresses=[...], seeds=[...]), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#15 0x00007f8b0ac8fd3b in fast_function (nk=<optimized out>, na=<optimized out>, n=0, pp_stack=0x7ffdaa02f3e0, 
    func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4783
#16 call_function (oparg=<optimized out>, pp_stack=0x7ffdaa02f3e0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
# gdb python3.5 coredump
GNU gdb (GDB) Fedora 7.11.1-75.fc24
...
Reading symbols from python3.5...Reading symbols from /usr/lib/debug/usr/bin/python3.5.debug...done.
done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/frens-jan/Workspaces/tgho/bndl/bndl/testvenv/bin/python -m bndl.util.supe'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f0a06657690 in ?? ()
[Current thread is 1 (Thread 0x7f09d7572700 (LWP 14413))]
Missing separate debuginfos, use: dnf debuginfo-install krb5-libs-1.14.3-8.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
(gdb) bt
#0  0x00007f0a06657690 in ?? ()
Python Exception <class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte: 
#1  0x00007f0a0624c63f in visit_decref (op=, data=<optimized out>)
    at /usr/src/debug/Python-3.5.1/Modules/gcmodule.c:373
#2  0x00007f0a0624b813 in subtract_refs (containers=<optimized out>)
    at /usr/src/debug/Python-3.5.1/Modules/gcmodule.c:398
#3  collect (generation=generation@entry=2, n_collected=n_collected@entry=0x7f09d75708c8, 
    n_uncollectable=n_uncollectable@entry=0x7f09d75708d0, nofail=nofail@entry=0)
    at /usr/src/debug/Python-3.5.1/Modules/gcmodule.c:951
#4  0x00007f0a0624c40d in collect_with_callback (generation=2) at /usr/src/debug/Python-3.5.1/Modules/gcmodule.c:1119
#5  0x00007f0a0624c811 in gc_collect (self=<optimized out>, args=<optimized out>, kws=<optimized out>)
    at /usr/src/debug/Python-3.5.1/Modules/gcmodule.c:1213
#6  0x00007f0a06199e39 in PyCFunction_Call (
    func=func@entry=<built-in method collect of module object at remote 0x7f09fa2f0318>, args=args@entry=(), 
    kwds=kwds@entry=0x0) at /usr/src/debug/Python-3.5.1/Objects/methodobject.c:98
#7  0x00007f0a06210dc7 in call_function (oparg=<optimized out>, pp_stack=0x7f09d7570a00)
    at /usr/src/debug/Python-3.5.1/Python/ceval.c:4684
#8  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f09f44fe868, for file /home/frens-jan/Workspaces/tgho/bndl/bndl/bndl/compute/shuffle.py, line 31, in collect (), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#9  0x00007f0a06210d3b in fast_function (nk=<optimized out>, na=<optimized out>, n=0, pp_stack=0x7f09d7570b40, 
    func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4783
#10 call_function (oparg=<optimized out>, pp_stack=0x7f09d7570b40) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
#11 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f09c8001058, for file /home/frens-jan/Workspaces/tgho/bndl/bndl/bndl/compute/shuffle.py, line 163, in _spill_bucket (self=<SortedBucket(id=('abvo8wka', 23, 10), comb=None, element_size=1009, disk_blocks=[], disk_container=<StorageContainerFactory(deserialize=<function at remote 0x7f09f8afa378>, container_cls=<type at remote 0x55cf4b9c30f8>, io_wrapper=<built-in function identity>, serialize=<function at remote 0x7f09f8afa2f0>, mode='b') at remote 0x7f09d7574198>, key=None, lock=<_thread.RLock at remote 0x7f09d75c2a50>, block_size_mb=4, memory_blocks=[<SerializedInMemory(id=('abvo8wka', 23, 10, '1.0'), data=<bytearray at remote 0x7f09d7587420>, provider=<StorageContainerFactory(deserialize=<function at remote 0x7f09f8afa378>, container_cls=<type at remote 0x55cf4b9c2508>, io_wrapper=<built-in function identity>, serialize=<function at remote 0x7f09f8afa2f0>, mode='b') at remote 0x7f09d7574160>) at remote 0x7f09d7587048>], batches=[[<...>]], memory_container=<...>) at remote 0x7f09d75e9e58>, disk=False...(truncated), 
    throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#12 0x00007f0a06212753 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, 
    locals=locals@entry=0x0, args=args@entry=0x7f09c8000fe0, argcount=3, kws=0x7f09c8000ff8, kwcount=0, 
    defs=0x7f09fda12bf8, defcount=1, kwdefs=0x0, closure=0x0, name='_spill_bucket', qualname='Bucket._spill_bucket')
    at /usr/src/debug/Python-3.5.1/Python/ceval.c:3995
#13 0x00007f0a0620ee19 in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, 
    pp_stack=0x7f09d7570d50, func=<optimized out>) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4793
#14 call_function (oparg=<optimized out>, pp_stack=0x7f09d7570d50) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4710
#15 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f09c8000e38, for file /home/frens-jan/Workspaces/tgho/bndl/bndl/bndl/compute/shuffle.py, line 194, in spill (self=<SortedBucket(id=('abvo8wka', 23, 10), comb=None, element_size=1009, disk_blocks=[], disk_container=<StorageContainerFactory(deserialize=<function at remote 0x7f09f8afa378>, container_cls=<type at remote 0x55cf4b9c30f8>, io_wrapper=<built-in function identity>, serialize=<function at remote 0x7f09f8afa2f0>, mode='b') at remote 0x7f09d7574198>, key=None, lock=<_thread.RLock at remote 0x7f09d75c2a50>, block_size_mb=4, memory_blocks=[<SerializedInMemory(id=('abvo8wka', 23, 10, '1.0'), data=<bytearray at remote 0x7f09d7587420>, provider=<StorageContainerFactory(deserialize=<function at remote 0x7f09f8afa378>, container_cls=<type at remote 0x55cf4b9c2508>, io_wrapper=<built-in function identity>, serialize=<function at remote 0x7f09f8afa2f0>, mode='b') at remote 0x7f09d7574160>) at remote 0x7f09d7587048>], batches=[[<...>]], memory_container=<...>) at remote 0x7f09d75e9e58>, disk=False, max_me...(truncated), 
    throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:3209
#16 0x00007f0a06212753 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, 
    locals=locals@entry=0x0, args=args@entry=0x7f09d75ed8d0, argcount=3, kws=kws@entry=0x7f0a0676f060, kwcount=0, 
    defs=0x7f09fa3263e0, defcount=2, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at /usr/src/debug/Python-3.5.1/Python/ceval.c:3995
#17 0x00007f0a06212833 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, 
    args=args@entry=0x7f09d75ed8d0, argcount=<optimized out>, kws=kws@entry=0x7f0a0676f060, kwcount=0, 
    defs=0x7f09fa3263e0, defcount=2, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.5.1/Python/ceval.c:4016
#18 0x00007f0a0617d545 in function_call (func=<function at remote 0x7f09f8029b70>, 
    arg=(<SortedBucket(id=('abvo8wka', 23, 10), comb=None, element_size=1009, disk_blocks=[], disk_container=<StorageContainerFactory(deserialize=<function at remote 0x7f09f8afa378>, container_cls=<type at remote 0x55cf4b9c30f8>, io_wrapp---Type <return> to continue, or q <return> to quit---q

as well as on Centos 7 with python 3.4.3

$ gdb python3.4 core.11602
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
...
Reading symbols from /home/frens.jan.rumph/venv/bin/python3.4...Reading symbols from /usr/lib/debug/usr/bin/python3.4m.debug...done.
done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/frens.jan.rumph/venv/bin/python -m bndl.util.supervisor.child bndl.comput'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fc26fa94a96 in subtype_dealloc (self=0x7fc2537e3d30) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:1157
1157            _PyObject_GC_UNTRACK(self);
(gdb) bt
#0  0x00007fc26fa94a96 in subtype_dealloc (self=<Task at remote 0x7fc2537e3d30>) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:1157
#1  0x00007fc26faf5c95 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0xf980e8, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/net/peer.py, line 302, in _serve (self=<RMIPeerNode(_iotasks=<WeakSet(_pending_removals=[], _remove=<function at remote 0x7fc261855b70>, _iterating=set(), data=set()) at remote 0x7fc2537f04a8>, name='nl.tgho.priv.sp-dev03.worker.11593.0.13', conn=<Connection(bytes_received=6423, bytes_sent=5608, reader=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 1...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1412
#2  0x00007fc26fa6195f in gen_send_ex (arg=<optimized out>, exc=0, gen=<optimized out>, gen=<optimized out>) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#3  0x00007fc26faea324 in builtin_next (self=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/bltinmodule.c:1195
#4  0x00007fc26faf7172 in call_function (oparg=<optimized out>, pp_stack=0x7fffa5e8b220) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4266
#5  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x11ca6c8, for file /usr/lib64/python3.4/asyncio/tasks.py, line 238, in _step (self=<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>, ...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#6  0x00007fc26faf7f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7fc2633aa1e0>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=3, kws=0x11ca638, kwcount=kwcount@entry=0,
    defs=0x7fc2633abda0, defcount=2, kwdefs=0x0, closure=(<cell at remote 0x7fc263619648>,)) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#7  0x00007fc26faf5a12 in fast_function (nk=0, na=3, n=<optimized out>, pp_stack=0x7fffa5e8b450, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#8  call_function (oparg=<optimized out>, pp_stack=0x7fffa5e8b450) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
#9  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x11ca488, for file /usr/lib64/python3.4/asyncio/tasks.py, line 292, in _wakeup (self=<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#10 0x00007fc26faf7f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7fc2633aa270>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fc2537ee220, argcount=2, kws=kws@entry=0x0, kwcount=kwcount@entry=0,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#11 0x00007fc26fa683be in function_call (func=<function at remote 0x7fc2633acf28>,
    arg=(<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>, 21: <SelectorKey at remote 0x7fc252f93518>, 22: <SelectorKey at remote 0x7fc2537f4258>, 2...(truncated), kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/funcobject.c:632
#12 0x00007fc26fa3fdcc in PyObject_Call (func=func@entry=<function at remote 0x7fc2633acf28>,
    arg=arg@entry=(<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>, 21: <SelectorKey at remote 0x7fc252f93518>, 22: <SelectorKey at remote 0x7fc2537f4258>, 2...(truncated), kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
#13 0x00007fc26faf37c9 in ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7fffa5e8b778, func=<function at remote 0x7fc2633acf28>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4590
#14 PyEval_EvalFrameEx (f=f@entry=Frame 0xf883e8, for file /usr/lib64/python3.4/asyncio/events.py, line 120, in _run (self=<Handle at remote 0x7fc1ebc25228>), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2902
#15 0x00007fc26faf6e40 in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7fffa5e8b8e0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4363
#16 call_function (oparg=<optimized out>, pp_stack=0x7fffa5e8b8e0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
#17 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xfc4d38, for file /usr/lib64/python3.4/asyncio/base_events.py, line 1172, in _run_once (self=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>, 21...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#18 0x00007fc26faf6e40 in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7fffa5e8ba50, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4363
#19 call_function (oparg=<optimized out>, pp_stack=0x7fffa5e8ba50) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
#20 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xf850c8, for file /usr/lib64/python3.4/asyncio/base_events.py, line 276, in run_forever (self=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7fc2618380b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7fc26184eba8>, _epoll=<select.epoll at remote 0x7fc270007eb8>, _fd_to_key={5: <SelectorKey at remote 0x7fc261831620>, 7: <SelectorKey at remote 0x7fc2618316d0>, 8: <SelectorKey at remote 0x7fc261831570>, 9: <SelectorKey at remote 0x7fc261831af0>, 10: <SelectorKey at remote 0x7fc261831bf8>, 11: <SelectorKey at remote 0x7fc2537ce0a0>, 12: <SelectorKey at remote 0x7fc2537ce048>, 13: <SelectorKey at remote 0x7fc2537dae08>, 14: <SelectorKey at remote 0x7fc2537ceb48>, 15: <SelectorKey at remote 0x7fc2537da150>, 16: <SelectorKey at remote 0x7fc2537ec678>, 17: <SelectorKey at remote 0x7fc2537ec7d8>, 18: <SelectorKey at remote 0x7fc2537daba0>, 19: <SelectorKey at remote 0x7fc2537dad00>, 20: <SelectorKey at remote 0x7fc2537ec728>, 2...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#21 0x00007fc26faf6e40 in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7fffa5e8bbc0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4363
#22 call_function (oparg=<optimized out>, pp_stack=0x7fffa5e8bbc0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
---Type <return> to continue, or q <return> to quit---
$ gdb python3.4 core.11623
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
...
Reading symbols from /home/frens.jan.rumph/venv/bin/python3.4...Reading symbols from /usr/lib/debug/usr/bin/python3.4m.debug...done.
done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/frens.jan.rumph/venv/bin/python -m bndl.util.supervisor.child bndl.comput'.
Program terminated with signal 11, Segmentation fault.
#0  _PyObject_Malloc (ctx=0x0, nbytes=56) at /usr/src/debug/Python-3.4.3/Objects/obmalloc.c:1159
1159                if ((pool->freeblock = *(block **)bp) != NULL) {
(gdb) bt
#0  _PyObject_Malloc (ctx=0x0, nbytes=56) at /usr/src/debug/Python-3.4.3/Objects/obmalloc.c:1159
#1  0x00007f2530ca90c9 in _PyObject_GC_Malloc (basicsize=basicsize@entry=32) at /usr/src/debug/Python-3.4.3/Modules/gcmodule.c:1725
#2  0x00007f2530c0b17a in PyType_GenericAlloc (type=0x21fdd78, nitems=0) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:916
#3  0x00007f2530c0f4f3 in type_call (type=0x21fdd78, args=(),
    kwds={'loop': <_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at remote 0x7f252096c888>, 21: <SelectorKey at remote 0x7f2520975308>, 22: <SelectorKey at remote 0x7f2520979678>, 23: <...(truncated)) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:883
#4  0x00007f2530bb9dcc in PyObject_Call (func=func@entry=<type at remote 0x21fdd78>, arg=arg@entry=(),
    kw=kw@entry={'loop': <_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at remote 0x7f252096c888>, 21: <SelectorKey at remote 0x7f2520975308>, 22: <SelectorKey at remote 0x7f2520979678>, 23: <...(truncated)) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
#5  0x00007f2530c6c902 in do_call (nk=<optimized out>, na=0, pp_stack=0x7ffefe5eb260, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4495
#6  call_function (oparg=<optimized out>, pp_stack=0x7ffefe5eb260) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4293
#7  PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f252004c648, for file /usr/lib64/python3.4/asyncio/streams.py, line 391, in _wait_for_data (self=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at ...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#8  0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f250a4fccf0, gen=0x7f250a4fccf0) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#9  0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f250a4fccf0, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#10 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f25200bd9d0, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/util/aio.py, line 129, in readexactly (self=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f2520...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#11 0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f250a4fc948, gen=0x7f250a4fc948) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#12 0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f250a4fc948, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#13 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f252004c448, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/net/connection.py, line 179, in _recv (self=<Connection(bytes_received=11390, bytes_sent=10841, reader=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at ...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#14 0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f250806e5e8, gen=0x7f250806e5e8) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#15 0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f250806e5e8, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#16 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x231ee48, for file /usr/lib64/python3.4/asyncio/tasks.py, line 358, in wait_for (fut=<generator at remote 0x7f250806e5e8>, timeout=None, loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#17 0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f250a4fc900, gen=0x7f250a4fc900) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#18 0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f250a4fc900, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#19 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x2390668, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/net/connection.py, line 208, in recv (self=<Connection(bytes_received=11390, bytes_sent=10841, reader=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#20 0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f25080bc870, gen=0x7f25080bc870) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#21 0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f25080bc870, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#22 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x2376988, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/net/peer.py, line 75, in recv (self=<RMIPeerNode(_iotasks=<WeakSet(_pending_removals=[], _remove=<function at remote 0x7f25200a2158>, _iterating=set(), data={<weakref at remote 0x7f250bfcc9a8>, <weakref at remote 0x7f25080d03b8>, <weakref at remote 0x7f25080b4ea8>, <weakref at remote 0x7f25081068b8>, <weakref at remote 0x7f250a4fd7c8>, <weakref at remote 0x7f2508106958>, <weakref at remote 0x7f250a08d318>, <weakref at remote 0x7f250a4ecd18>, <weakref at remote 0x7f252000b548>, <weakref at remote 0x7f25200acb38>, <weakref at remote 0x7f250bf1d818>, <weakref at remote 0x7f250806bf98>}) at remote 0x7f25200840f0>, name='nl.tgho.priv.sp-dev03.worker.11593.0.56', conn=<Connection(bytes_received=11390, bytes_sent=10841, reader=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#23 0x00007f2530bdb95f in gen_send_ex (arg=arg@entry=None, exc=exc@entry=0, gen=0x7f25080bc900, gen=0x7f25080bc900) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#24 0x00007f2530bdbe62 in _PyGen_Send (gen=gen@entry=0x7f25080bc900, arg=arg@entry=None) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:158
#25 0x00007f2530c70792 in PyEval_EvalFrameEx (
    f=f@entry=Frame 0x237d1b8, for file /home/frens.jan.rumph/venv/lib/python3.4/site-packages/bndl/net/peer.py, line 279, in _serve (self=<RMIPeerNode(_iotasks=<WeakSet(_pending_removals=[], _remove=<function at remote 0x7f25200a2158>, _iterating=set(), data={<weakref at remote 0x7f250bfcc9a8>, <weakref at remote 0x7f25080d03b8>, <weakref at remote 0x7f25080b4ea8>, <weakref at remote 0x7f25081068b8>, <weakref at remote 0x7f250a4fd7c8>, <weakref at remote 0x7f2508106958>, <weakref at remote 0x7f250a08d318>, <weakref at remote 0x7f250a4ecd18>, <weakref at remote 0x7f252000b548>, <weakref at remote 0x7f25200acb38>, <weakref at remote 0x7f250bf1d818>, <weakref at remote 0x7f250806bf98>}) at remote 0x7f25200840f0>, name='nl.tgho.priv.sp-dev03.worker.11593.0.56', conn=<Connection(bytes_received=11390, bytes_sent=10841, reader=<StreamReader(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at rem...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:1932
#26 0x00007f2530bdb95f in gen_send_ex (arg=<optimized out>, exc=0, gen=<optimized out>, gen=<optimized out>) at /usr/src/debug/Python-3.4.3/Objects/genobject.c:104
#27 0x00007f2530c64324 in builtin_next (self=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/bltinmodule.c:1195
#28 0x00007f2530c71172 in call_function (oparg=<optimized out>, pp_stack=0x7ffefe5ebe70) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4266
#29 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7f25070991d8, for file /usr/lib64/python3.4/asyncio/tasks.py, line 238, in _step (self=<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at remote 0x7f252096c8...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#30 0x00007f2530c71f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7f25245241e0>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=3, kws=0x23f7b58, kwcount=kwcount@entry=0, defs=0x7f2524525da0, defcount=2, kwdefs=0x0,
    closure=(<cell at remote 0x7f2524793648>,)) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#31 0x00007f2530c6fa12 in fast_function (nk=0, na=3, n=<optimized out>, pp_stack=0x7ffefe5ec0a0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#32 call_function (oparg=<optimized out>, pp_stack=0x7ffefe5ec0a0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
#33 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x23f79a8, for file /usr/lib64/python3.4/asyncio/tasks.py, line 292, in _wakeup (self=<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at remote 0x7f252096c888>...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#34 0x00007f2530c71f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7f2524524270>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f25200b9f20, argcount=2, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=0x0)
    at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#35 0x00007f2530be23be in function_call (func=<function at remote 0x7f2524526f28>,
    arg=(<Task(_loop=<_UnixSelectorEventLoop(_current_handle=None, _ready=<collections.deque at remote 0x7f25229b20b0>, _closed=False, _selector=<EpollSelector(_map=<_SelectorMapping(_selector=<...>) at remote 0x7f25229c8ba8>, _epoll=<select.epoll at remote 0x7f2531181eb8>, _fd_to_key={5: <SelectorKey at remote 0x7f25229ab620>, 7: <SelectorKey at remote 0x7f25229ab6d0>, 8: <SelectorKey at remote 0x7f25229ab570>, 9: <SelectorKey at remote 0x7f25209552b0>, 10: <SelectorKey at remote 0x7f25209553b8>, 11: <SelectorKey at remote 0x7f25209554c0>, 12: <SelectorKey at remote 0x7f2520955620>, 13: <SelectorKey at remote 0x7f2520955780>, 14: <SelectorKey at remote 0x7f25209558e0>, 15: <SelectorKey at remote 0x7f2520955a40>, 16: <SelectorKey at remote 0x7f2520955ba0>, 17: <SelectorKey at remote 0x7f2520955eb8>, 18: <SelectorKey at remote 0x7f25201069e8>, 19: <SelectorKey at remote 0x7f252096c048>, 20: <SelectorKey at remote 0x7f252096c888>, 21: <SelectorKey at remote 0x7f2520975308>, 22: <SelectorKey at remote 0x7f2520979678>, 2...(truncated), kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/funcobject.c:632
#36 0x00007f2530bb9dcc in PyObject_Call (func=func@entry=<function at remote 0x7f2524526f28>,
---Type <return> to continue, or q <return> to quit---

(sorry for the large dumps)

frensjan commented 7 years ago

And also saw a SIGBUS:

$ gdb python3.4 core.8455
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
...
Reading symbols from /usr/bin/python3.4...Reading symbols from /usr/lib/debug/usr/bin/python3.4m.debug...done.
done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /home/frens.jan.rumph/venv/lib/python3.4/site-packages/numpy/core/../.libs/libgfortran-ed201abd.so.3.0.0
Core was generated by `/home/frens.jan.rumph/venv/bin/python -m bndl.util.supervisor.child bndl.comput'.
Program terminated with signal 7, Bus error.
#0  malloc_printerr (ar_ptr=0x100101f0100101a, ptr=0x7f067955da60 <generations+32>, str=0x7f06785a2b8c "free(): invalid size", action=3) at malloc.c:5009
5009        set_arena_corrupt (ar_ptr);
(gdb) bt
#0  malloc_printerr (ar_ptr=0x100101f0100101a, ptr=0x7f067955da60 <generations+32>, str=0x7f06785a2b8c "free(): invalid size", action=3) at malloc.c:5009
#1  _int_free (av=0x100101f0100101a, p=<optimized out>, have_lock=0) at malloc.c:3842
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#2  0x00007f067921cf14 in unpack_iterable (v=v@entry=, argcnt=0, argcntafter=<optimized out>, sp=<optimized out>, sp@entry=0x7f0665e357d0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3866
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#3  0x00007f0679221b54 in PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2130
#4  0x00007f0679227f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7f066b9a1930>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f0665dd5d98, argcount=3, kws=kws@entry=0x0, kwcount=kwcount@entry=0,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=(<cell at remote 0x7f066b0b6d08>,)) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#5  0x00007f06791983be in function_call (func=<function at remote 0x7f066b0d56a8>, arg=, kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/funcobject.c:632
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#6  0x00007f067916fdcc in PyObject_Call (func=func@entry=<function at remote 0x7f066b0d56a8>, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#7  0x00007f0679184ead in method_call (func=<function at remote 0x7f066b0d56a8>, arg=, kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/classobject.c:347
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#8  0x00007f067916fdcc in PyObject_Call (func=func@entry=<method at remote 0x7f0601772f88>, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#9  0x00007f06791c7a90 in slot_tp_init (self=<optimized out>, args=, kwds=0x0) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:6177
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#10 0x00007f06791c5536 in type_call (type=<optimized out>, args=, kwds=0x0) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:898
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#11 0x00007f067916fdcc in PyObject_Call (func=func@entry=<type at remote 0x1f98b78>, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
#12 0x00007f0679222902 in do_call (nk=<optimized out>, na=2, pp_stack=0x7f0653a76c90, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4495
#13 call_function (oparg=<optimized out>, pp_stack=0x7f0653a76c90) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4293
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#14 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#15 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7f066b99ddb0>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f0665dd6360, argcount=2, kws=kws@entry=0x0, kwcount=kwcount@entry=0,
    defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#16 0x00007f06791983be in function_call (func=<function at remote 0x7f066b0d3c80>, arg=, kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/funcobject.c:632
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#17 0x00007f067916fdcc in PyObject_Call (func=func@entry=<function at remote 0x7f066b0d3c80>, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#18 0x00007f0679184ead in method_call (func=<function at remote 0x7f066b0d3c80>, arg=, kw=0x0) at /usr/src/debug/Python-3.4.3/Objects/classobject.c:347
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#19 0x00007f067916fdcc in PyObject_Call (func=func@entry=<method at remote 0x7f060179c048>, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#20 0x00007f06791c7dc0 in slot_tp_call (self=<optimized out>, args=, kwds=0x0) at /usr/src/debug/Python-3.4.3/Objects/typeobject.c:5962
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#21 0x00007f067916fdcc in PyObject_Call (func=func@entry=, arg=arg@entry=, kw=kw@entry=0x0) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
#22 0x00007f0679222902 in do_call (nk=<optimized out>, na=1, pp_stack=0x7f0653a770d0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4495
#23 call_function (oparg=<optimized out>, pp_stack=0x7f0653a770d0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4293
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#24 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#25 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7f066c5886f0>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=3, kws=0x7f0568153318, kwcount=kwcount@entry=0,
    defs=0x7f067114f290, defcount=1, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#26 0x00007f0679225a12 in fast_function (nk=0, na=3, n=<optimized out>, pp_stack=0x7f0653a77300, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#27 call_function (oparg=<optimized out>, pp_stack=0x7f0653a77300) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#28 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#29 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7f066c588780>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f060175dfa0, argcount=2, kws=kws@entry=0x7f067976f060, kwcount=kwcount@entry=0,
    defs=defs@entry=0x7f0671171ba0, defcount=defcount@entry=2, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#30 0x00007f06791984b3 in function_call (func=<function at remote 0x7f06044567b8>, arg=, kw={}) at /usr/src/debug/Python-3.4.3/Objects/funcobject.c:632
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#31 0x00007f067916fdcc in PyObject_Call (func=func@entry=<function at remote 0x7f06044567b8>, arg=arg@entry=, kw=kw@entry={}) at /usr/src/debug/Python-3.4.3/Objects/abstract.c:2040
#32 0x00007f06792237c9 in ext_do_call (nk=<optimized out>, na=1, flags=<optimized out>, pp_stack=0x7f0653a77628, func=<function at remote 0x7f06044567b8>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4590
#33 PyEval_EvalFrameEx (f=f@entry=<super at remote 0x22bd748>, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2902
#34 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7f066c588d20>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x7f06600ee710, kwcount=kwcount@entry=0,
    defs=0x0, defcount=0, kwdefs=0x0, closure=(<cell at remote 0x7f066bd20cd8>,)) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#35 0x00007f0679225a12 in fast_function (nk=0, na=2, n=<optimized out>, pp_stack=0x7f0653a77850, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#36 call_function (oparg=<optimized out>, pp_stack=0x7f0653a77850) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#37 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#38 0x00007f0679226e40 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0x7f0653a779c0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4363
#39 call_function (oparg=<optimized out>, pp_stack=0x7f0653a779c0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#40 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#41 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7f067116f030>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x7f06666263b8, kwcount=kwcount@entry=0,
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#42 0x00007f0679225a12 in fast_function (nk=0, na=2, n=<optimized out>, pp_stack=0x7f0653a77bf0, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#43 call_function (oparg=<optimized out>, pp_stack=0x7f0653a77bf0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#44 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#45 0x00007f0679226e40 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0x7f0653a77d60, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4363
#46 call_function (oparg=<optimized out>, pp_stack=0x7f0653a77d60) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#47 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#48 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=_co@entry=<code at remote 0x7f067116f030>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x7f05e4001938, kwcount=kwcount@entry=0,
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:3617
#49 0x00007f0679225a12 in fast_function (nk=0, na=2, n=<optimized out>, pp_stack=0x7f0653a77f90, func=<optimized out>) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4373
#50 call_function (oparg=<optimized out>, pp_stack=0x7f0653a77f90) at /usr/src/debug/Python-3.4.3/Python/ceval.c:4291
Python Exception <type 'exceptions.RuntimeError'> Type does not have a target.:
#51 PyEval_EvalFrameEx (f=f@entry=, throwflag=throwflag@entry=0) at /usr/src/debug/Python-3.4.3/Python/ceval.c:2862
#52 0x00007f0679227f3e in PyEval_EvalCodeEx (_co=<code at remote 0x7f066b992300>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7f060400b9f0, argcount=3, kws=kws@entry=0x7f067976f060, kwcount=kwcount@entry=0,
---Type <return> to continue, or q <return> to quit---
Martiusweb commented 7 years ago

Every time I encountered those suspicious segfaults and weird behavior (AttributeError on an object of the wrong type, etc), it was related to an unsafe use of threads.

I am not an expert of cpython, but I don't think those traces will reveal anything useful, especially not a bug in python or asyncio.

You should look in your code when an how threads are used, and avoid any operation on objects manipulated by the loop without synchronization (except the one and only loop.call_soon_threadsafe).

Also, in your first message you say:

The lock serializes multiple readers on the same connection. I'm using a lock because the asynchronous code is interacting with threaded code.

but you're using an asyncio.Lock object, which will not provide thread-level locking: it only allow synchronization between coroutines running on the same loop. Are you sure that you're using this lock correctly?

frensjan commented 7 years ago

Thanks @Martiusweb. I think that this is the most likely hypothesis. I'll close the issue.

For the record: The threaded part of the application serializes access to certain objects using threading.RLock objects. The interaction between the threaded side of the application and the asyncio event loop side is 'serialized' by asyncio.call_soon_threadsafe which schedules a callback in the event queue (and writes to self to get the selector going ...); the scheduled callback subsequently runs a co-routine to do the actual asyncio work (writing a framed message on a socket).

From the back traces from GDB it seems like GC pointers are mutilated. I'm able to reproduce the issue under CPU core contention (running many processes concurrently which issue messages among each other). This leads me to think that indeed this has to do with a race condition somewhere.

That being said: my code doesn't release the GIL at any point so I'd expect any thread safety issues to be limited to python level race conditions, and not raise segfaults in code such as:

#define _PyObject_GC_TRACK(o) do { \
    PyGC_Head *g = _Py_AS_GC(o); \
    if (_PyGCHead_REFS(g) != _PyGC_REFS_UNTRACKED) \
        Py_FatalError("GC object already tracked"); \
    _PyGCHead_SET_REFS(g, _PyGC_REFS_REACHABLE); \
    g->gc.gc_next = _PyGC_generation0; \
    g->gc.gc_prev = _PyGC_generation0->gc.gc_prev; \
    g->gc.gc_prev->gc.gc_next = g; \
    _PyGC_generation0->gc.gc_prev = g; \
    } while (0);
frensjan commented 7 years ago

For the record: I've been able to trace the SIGSEGV's to issue 26617 in CPython which is fixed in 7bfa6f2.

I've check this by reproducing the issue (running into SIGSEGV) with a build of the commit just before 7bfa6f2 and not being able to reproduce it with a build of commit 7bfa6f2.

I'm going out on a limb here, but I hold it possible that the original (weird and seemingly incorrect) tracebacks are caused by this issue as well. Fingers crossed.