A desktop application generates information in a QThread and signals this infos
to the Main thread of the widget. The signaling crashes the application at
random locations and sometimes overwrites readonly memory.
Qt Promises this is a safe way to send information between threads.
On Linux this works as expected.
The example code has been stripped to signal only strings from the thread to
the widget. The widget will only read the signaled message (copy it to a new
buffer) and does nothing complicated/gui related with the data.
virtualenv ./var/venv/pyside
# then copy /Library/Python/2.7/site-packages/Pyside to the
virtualenv/lib/python2.7/site-packages
also crashes without a virtualenv
Steps
run this test multiple times to about 5000 dispatched events
usually happens with 100, but sometimes later
while [ 1 ]; do python ./qt-signal-bug-test.py 5000 || sleep 20; done
Expectation
The program displays a window and uses all available cpu due to the
dispatching of the events.
Actual
Sometimes the program crashes. The location and exception is not reproducible.
Sometimes the program ends in a non responsive way
Sometimes readonly memory (emitted data) is overwritten (with print on, 10x10
items):
# -*- coding: utf-8 -*-
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
'''
App that displays a GUI that uses a model that is updated using
a backend.
Running this app displays how many (hundred) events have been dispatched
between the backend-thread and the GUI.
Please also read the bug report at the end of the code
In real world the backend is a tornado ioloop that requests network resources
and the GUI should process JSON from those requests. Inspired by backbone.js.
The workaround is not to use threads in pyside.
'''
import atexit
import os
import time
import sys
from PySide.QtGui import QApplication, QWidget, QMainWindow, QVBoxLayout
from PySide.QtCore import (
Qt, QLocale, QTranslator, QTimer, QThread,
QObject, Signal, QCoreApplication
)
LOOP_TIMER_NOOP = 0.01
class many_items(QWidget):
'''
Widget that exists only to be visible and to process the backend data.
'''
def __init__(self, parent, dispatcher):
QWidget.__init__(self, parent)
dispatcher.model_changed.connect(self.handler)
self.event_handled = 0
def handler(self, model_json):
# process data from the backend thread
data = ''
for c in model_json:
data += str(c)
# print 'many_item', data, self.event_handled
self.event_handled += 1
class some_items(QWidget):
'''
Widget that contains multiple widgets connected to the backend data.
The widget itself is also connected, it should be able to have many
more 'many_items' in the QVBoxLayout
'''
def __init__(self, parent, dispatcher):
QWidget.__init__(self, parent)
self.ui = QVBoxLayout(self)
self.items = []
self.event_handled = 0
dispatcher.model_changed.connect(self.handler)
for i in range(0, 10):
item = many_items(self, dispatcher)
self.ui.addWidget(item)
self.items.append(item)
def handler(self, model_json):
# process data from the backend thread
data = ''
for c in model_json:
data += str(c)
# print 'some_item', data, self.event_handled
self.event_handled += 1
class main_item(QWidget):
'''
Main item that is used as central widget and contains many some_items in a
QVBoxLayout.
The widget itself is also connected.
'''
def __init__(self, parent, dispatcher):
QWidget.__init__(self, parent)
self.resize(30, 60)
self.ui = QVBoxLayout(self)
self.items = []
self.event_handled = 0
dispatcher.model_changed.connect(self.handler)
# Reduce the range to simplify the testcase and decrease the number of
# connected objects.
# somexmany: dispatched events required
# 0-10x10: ~ 1k
# 100x100: > 5k
# 10x100: > 20k
# 100x10: ~ 1k
# 1x1: > 1M
# 10x1: < 1k
# 1000x1: > 5k
# if >5k: killall python, restart!
for i in range(0, 10):
item = some_items(self, dispatcher)
self.ui.addWidget(item)
self.items.append(item)
def handler(self, model_json):
# process data from the backend thread
data = ''
for c in model_json:
data += str(c)
# print 'main_item', data, self.event_handled
self.event_handled += 1
#
# the following code is required to make the window and simulate a backend
#
class mainwindow(QMainWindow):
def __init__(self, backend, logger=None):
QMainWindow.__init__(self)
self.backend = backend
self.setCentralWidget(main_item(self, self.backend))
self.resize(300, 600)
self.show()
self.backend.shutdown.connect(self.shutdown)
self.backend.request_data.emit()
def shutdown(self):
print 'mainwindow.shutdown'
QApplication.setQuitOnLastWindowClosed(True)
self.close()
class backend_thread(QThread):
'''
Backend Thread that is responsible to create the backend dispatcher and
make it available to the main-thread. The Backend Thread continuously
executes QCoreApplication.processEvents to ensure that emit'ed signals are
received in this thread. The dispatcher QObject is created when the thread
has started.
'''
def __init__(self, dispatcher_class):
QThread.__init__(self, None)
self.backend_dispatcher_class = dispatcher_class
self.backend_dispatcher = None
def get_dispatcher(self):
'''
Returns the dispatcher for the main-thread.
Should be called after start()
'''
while self.backend_dispatcher is None:
# Thread not yet running
# wait until the dispatcher is available
time.sleep(LOOP_TIMER_NOOP)
return self.backend_dispatcher
def run(self):
'''
Backend Thread main. Schedules qt event loop and enters the tornado
event loop.
'''
self.backend_dispatcher = self.backend_dispatcher_class()
self.backend_dispatcher.shutdown.connect(self.quit)
self.exec_()
class backend_dispatcher(QObject):
'''
Backend Dispatcher to be used by the main thread. All functions of this
class are private and must not be called from the main thread. only
signals may be emitted that qt promises to be thread safe. Most signals
receive a done_signal and a failed_signal in order to handle the async
requests.
'''
model_changed = Signal(basestring)
request_data = Signal()
available = Signal()
shutdown = Signal()
def __init__(self):
QObject.__init__(self)
# receive signal from main thread
self.request_data.connect(self._request_data)
self.dispatched_events = 0
def _request_data(self):
# send response to main thread
# the data is chosen to see the actual corruption
# the corruption is not consistently happening
self.model_changed.emit(
'1234567812345678'
'1234pay8load5678'
'1234567812345678'
'1234567812345678'
'')
self._schedule_request_data()
def _schedule_request_data(self):
'''
Schedule request_data without producing a endless stack.
'''
self.dispatched_events += 1
if self.dispatched_events % 100 == 0:
print self.dispatched_events, int(sys.argv[1])
if int(sys.argv[1]) - self.dispatched_events < 0:
print 'requested amount of requests reached'
self.shutdown.emit()
# do not schedule more requests
# sys.exit(0)
else:
timer = QTimer(self)
timer.start(10)
timer.setSingleShot(True)
timer.timeout.connect(self._request_data)
def run_gui():
'''
Run the GUI for the application.
'''
qApp = QApplication(sys.argv)
backend = backend_thread(backend_dispatcher)
backend.start()
dispatcher = backend.get_dispatcher()
item_window = mainwindow(dispatcher)
item_window.show()
exit_code = qApp.exec_()
sys.exit(exit_code)
if __name__ == '__main__':
run_gui()
'''
Bug Title: Signals between QThread and QWidget cause SIGSEGV in random location
Bug Description:
A desktop application generates information in a QThread and signals this infos
to the Main thread of the widget. The signaling crashes the application at
random locations and sometimes overwrites readonly memory.
Qt Promises this is a safe way to send information between threads.
On Linux this works as expected.
The example code has been stripped to signal only strings from the thread to
the widget. The widget will only read the signaled message (copy it to a new
buffer) and does nothing complicated/gui related with the data.
Pyside Version: 1.2.4
Precondition
linux:
virtualenv ./var/venv/pyside
source ./var/venv/pyside/bin/activate
pip install pyside
on a clean osx 10.11.6 do:
sudo xcode-select --install
brew install qt cmake
sudo pip install pyside virtualenv
then fix the installation, because it has broken @rpath and relative library
refs, as root:::
install_name_tool -change \
@rpath/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libpyside-python2.7.1.2.dylib
install_name_tool -change \
@rpath/libpyside-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libpyside-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/QtGui.so
install_name_tool -change \
@rpath/libpyside-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libpyside-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/QtCore.so
install_name_tool -change \
@rpath/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/QtGui.so
install_name_tool -change \
@rpath/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/libshiboken-python2.7.1.2.dylib \
/Library/Python/2.7/site-packages/PySide/QtCore.so
virtualenv ./var/venv/pyside
# then copy /Library/Python/2.7/site-packages/Pyside to the
virtualenv/lib/python2.7/site-packages
Steps:
# run this test multiple times to about 5000 dispatched events
# usually happens with 100, but sometimes later
while [ 1 ]; do python ./qt-signal-bug-test.py 5000 || sleep 20; done
Expectation:
The program displays a window and uses all available cpu due to the
dispatching of the events.
Actual:
Sometimes the program crashes. The location and exception is not reproducible.
Sometimes the program ends in a nonresponsive way
Sometimes readonly memory (emitted data) is overwritten (with print on, 10x10
items):
some_item 12345678123456781234pay8load56781234567812345678123456781234 10
some_item <�������� <�������� < 10
Also happens when
- No data emit('')
- Single connect in gui thread (but later / harder to reproduce)
- Qt and Tornado implementation
The loop of the steps also shows other weird exceptions like
Fatal Python error: deletion of interned string failed
Abort trap: 6
or
Process: python [34397]
Identifier: python
Version: 94
Code Type: X86-64 (Native)
Date/Time: 2016-08-07
OS Version: Mac OS X 10.11.6 (15G31)
Report Version: 11 (stripped)
Time Awake Since Boot: 380000 seconds
Time Since Wake: 14000 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000000000a8
VM Regions Near 0xa8:
-->
__TEXT 000000010a687000-000000010a688000 [ 4K] r-x/rwx SM=COW /data/*
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 org.python.python 0x000000010a693b82 PyObject_GetIter + 14
1 org.python.python 0x000000010a70da70 PyEval_EvalFrameEx + 1213
2 org.python.python 0x000000010a70d3c1 PyEval_EvalCodeEx + 1583
3 org.python.python 0x000000010a6b22e5 0x10a68a000 + 164581
4 org.python.python 0x000000010a694202 PyObject_Call + 99
5 org.python.python 0x000000010a69eff5 0x10a68a000 + 86005
6 org.python.python 0x000000010a694202 PyObject_Call + 99
7 org.python.python 0x000000010a712e83 PyEval_CallObjectWithKeywords + 165
8 libpyside-python2.7.1.2.dylib 0x000000010b236ae6 PySide::SignalManager::callPythonMetaMethod(QMetaMethod const&, void**, _object*, bool) + 598
9 libpyside-python2.7.1.2.dylib 0x000000010b2366ef PySide::SignalManager::qt_metacall(QObject*, QMetaObject::Call, int, void**) + 527
10 QtCore 0x000000010b39bca7 QObject::event(QEvent*) + 619
11 QtGui 0x000000010cc7b799 QWidget::event(QEvent*) + 3509
12 QtGui.so 0x000000010c50e1dd QWidgetWrapper::event(QEvent*) + 237
13 QtGui 0x000000010cc3855e QApplicationPrivate::notify_helper(QObject*, QEvent*) + 194
14 QtGui 0x000000010cc3aeb4 QApplication::notify(QObject*, QEvent*) + 6146
15 QtGui.so 0x000000010c002e2b QApplicationWrapper::notify(QObject*, QEvent*) + 267
16 QtCore 0x000000010b38b4f6 QCoreApplication::notifyInternal(QObject*, QEvent*) + 118
17 QtCore 0x000000010b38be69 QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) + 599
18 com.apple.CoreFoundation 0x00007fff92be6881 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
19 com.apple.CoreFoundation 0x00007fff92bc5fbc __CFRunLoopDoSources0 + 556
20 com.apple.CoreFoundation 0x00007fff92bc54df __CFRunLoopRun + 927
21 com.apple.CoreFoundation 0x00007fff92bc4ed8 CFRunLoopRunSpecific + 296
22 com.apple.HIToolbox 0x00007fff9419f935 RunCurrentEventLoopInMode + 235
23 com.apple.HIToolbox 0x00007fff9419f76f ReceiveNextEventCommon + 432
24 com.apple.HIToolbox 0x00007fff9419f5af _BlockUntilNextEventMatchingListInModeWithFilter + 71
25 com.apple.AppKit 0x00007fff944f5df6 _DPSNextEvent + 1067
26 com.apple.AppKit 0x00007fff944f5226 -[NSApplication _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 454
27 com.apple.AppKit 0x00007fff944e9d80 -[NSApplication run] + 682
28 QtGui 0x000000010cbf5ace QEventDispatcherMac::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 1496
29 QtCore 0x000000010b388bc7 QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 77
30 QtCore 0x000000010b388d41 QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 365
31 QtCore 0x000000010b38b94b QCoreApplication::exec() + 199
32 QtGui.so 0x000000010c004bef Sbk_QApplicationFunc_exec_(_object*) + 31
33 org.python.python 0x000000010a710a90 PyEval_EvalFrameEx + 13533
34 org.python.python 0x000000010a713541 0x10a68a000 + 562497
35 org.python.python 0x000000010a71030c PyEval_EvalFrameEx + 11609
36 org.python.python 0x000000010a70d3c1 PyEval_EvalCodeEx + 1583
37 org.python.python 0x000000010a70cd8c PyEval_EvalCode + 54
38 org.python.python 0x000000010a72ca42 0x10a68a000 + 666178
39 org.python.python 0x000000010a72cae5 PyRun_FileExFlags + 133
40 org.python.python 0x000000010a72c634 PyRun_SimpleFileExFlags + 698
41 org.python.python 0x000000010a73e011 Py_Main + 3137
42 libdyld.dylib 0x00007fff8c88a5ad start + 1
Thread 1:: Dispatch queue: com.apple.libdispatch-manager
0 libsystem_kernel.dylib 0x00007fff93321efa kevent_qos + 10
1 libdispatch.dylib 0x00007fff8899c165 _dispatch_mgr_invoke + 216
2 libdispatch.dylib 0x00007fff8899bdcd _dispatch_mgr_thread + 52
Thread 2:
0 libsystem_kernel.dylib 0x00007fff933215e2 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff91aea578 _pthread_wqthread + 1283
2 libsystem_pthread.dylib 0x00007fff91ae8341 start_wqthread + 13
Thread 3:
0 libsystem_kernel.dylib 0x00007fff933215e2 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff91aea578 _pthread_wqthread + 1283
2 libsystem_pthread.dylib 0x00007fff91ae8341 start_wqthread + 13
Thread 4:
0 libsystem_kernel.dylib 0x00007fff933215e2 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff91aea578 _pthread_wqthread + 1283
2 libsystem_pthread.dylib 0x00007fff91ae8341 start_wqthread + 13
Thread 5:
0 libsystem_kernel.dylib 0x00007fff933215e2 __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff91aea578 _pthread_wqthread + 1283
2 libsystem_pthread.dylib 0x00007fff91ae8341 start_wqthread + 13
Thread 6:: backend_thread
0 libsystem_kernel.dylib 0x00007fff93320db6 __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff91aeb728 _pthread_cond_wait + 767
2 org.python.python 0x000000010a73c1a4 PyThread_acquire_lock + 101
3 org.python.python 0x000000010a70cb0a PyEval_RestoreThread + 62
4 org.python.python 0x000000010a72b4a7 PyGILState_Ensure + 93
5 libshiboken-python2.7.1.2.dylib 0x000000010b263c7b Shiboken::GilState::GilState() + 27
6 QtGui.so 0x000000010c002d41 QApplicationWrapper::notify(QObject*, QEvent*) + 33
7 QtCore 0x000000010b38b4f6 QCoreApplication::notifyInternal(QObject*, QEvent*) + 118
8 QtCore 0x000000010b3b486f QTimerInfoList::activateTimers() + 751
9 QtCore 0x000000010b3b4f79 QEventDispatcherUNIX::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 347
10 QtCore 0x000000010b388bc7 QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 77
11 QtCore 0x000000010b388d41 QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 365
12 QtCore 0x000000010b2a81f5 QThread::exec() + 209
13 QtCore.so 0x000000010b073097 Sbk_QThreadFunc_exec_(_object*) + 87
14 org.python.python 0x000000010a710a90 PyEval_EvalFrameEx + 13533
15 org.python.python 0x000000010a70d3c1 PyEval_EvalCodeEx + 1583
16 org.python.python 0x000000010a6b22e5 0x10a68a000 + 164581
17 org.python.python 0x000000010a694202 PyObject_Call + 99
18 org.python.python 0x000000010a69eff5 0x10a68a000 + 86005
19 org.python.python 0x000000010a694202 PyObject_Call + 99
20 QtCore.so 0x000000010b07230a QThreadWrapper::run() + 90
21 QtCore 0x000000010b2a9bee QThreadPrivate::start(void*) + 386
22 libsystem_pthread.dylib 0x00007fff91aea99d _pthread_body + 131
23 libsystem_pthread.dylib 0x00007fff91aea91a _pthread_start + 168
24 libsystem_pthread.dylib 0x00007fff91ae8351 thread_start + 13
Thread 7:: com.apple.NSEventThread
0 libsystem_kernel.dylib 0x00007fff9331af72 mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff9331a3b3 mach_msg + 55
2 com.apple.CoreFoundation 0x00007fff92bc61c4 __CFRunLoopServiceMachPort + 212
3 com.apple.CoreFoundation 0x00007fff92bc568c __CFRunLoopRun + 1356
4 com.apple.CoreFoundation 0x00007fff92bc4ed8 CFRunLoopRunSpecific + 296
5 com.apple.AppKit 0x00007fff9464bd95 _NSEventThread + 149
6 libsystem_pthread.dylib 0x00007fff91aea99d _pthread_body + 131
7 libsystem_pthread.dylib 0x00007fff91aea91a _pthread_start + 168
8 libsystem_pthread.dylib 0x00007fff91ae8351 thread_start + 13
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x000000010ab769f0 rcx: 0x000000010a712738 rdx: 0x0000000000000000
rdi: 0x000000010ab769f0 rsi: 0x000000000000000c rbp: 0x00007fff55575bf0 rsp: 0x00007fff55575be0
r8: 0x000000010ab7c2b0 r9: 0x0000000000000000 r10: 0x00000000265e49dd r11: 0x00007fc484c00000
r12: 0x000000010ab9a4e1 r13: 0x000000010ade1d30 r14: 0x000000010ab769f0 r15: 0x0000000000000000
rip: 0x000000010a693b82 rfl: 0x0000000000010213 cr2: 0x00000000000000a8
Logical CPU: 0
Error Code: 0x00000004
Trap Number: 14
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 1
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 20489155
thread_create: 0
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=246.8M resident=0K(0%) swapped_out_or_unallocated=246.8M(100%)
Writable regions: Total=94.2M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=94.2M(100%)
VIRTUAL REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Accelerate.framework 128K 2
Activity Tracing 2048K 2
CG backing stores 1044K 2
CG image 16K 4
CG shared images 176K 6
CoreAnimation 64K 10
CoreUI image data 536K 7
CoreUI image file 192K 4
Dispatch continuations 8192K 2
Kernel Alloc Once 8K 3
MALLOC 48.1M 24
MALLOC guard page 32K 7
Memory Tag 242 12K 2
STACK GUARD 56.0M 9
Stack 11.1M 10
VM_ALLOCATE 6960K 17
__DATA 18.5M 224
__IMAGE 528K 2
__LINKEDIT 101.8M 44
__TEXT 145.1M 225
__UNICODE 552K 2
mapped file 112.2M 49
shared memory 16.3M 10
=========== ======= =======
TOTAL 529.0M 644
Model: MacBookPro8,1, 2 processors, Intel Core i5, 2,3 GHz, 8 GB, SMC 1.68f99
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In
'''
A desktop application generates information in a QThread and signals this infos to the Main thread of the widget. The signaling crashes the application at random locations and sometimes overwrites readonly memory. Qt Promises this is a safe way to send information between threads. On Linux this works as expected. The example code has been stripped to signal only strings from the thread to the widget. The widget will only read the signaled message (copy it to a new buffer) and does nothing complicated/gui related with the data.
PySide Version: 1.2.4
Precondition
on a clean osx 10.11.6 do:
then fix the installation, because it has broken @rpath and relative library refs, as root:::
also crashes without a virtualenv
Steps
Expectation
The program displays a window and uses all available cpu due to the dispatching of the events.
Actual
Sometimes the program crashes. The location and exception is not reproducible. Sometimes the program ends in a non responsive way Sometimes readonly memory (emitted data) is overwritten (with print on, 10x10 items):
More details at the end of the code.
Code to reproduce