ctb
commented
11 years ago On Thu, Feb 28, 2013 at 05:38:40PM -0800, Eric McDonald wrote:
We have various places where we wish to notify the user of progress being made during processing. In the original single-threaded model, callbacks were made from the C++ code doing the heavy-lifting to Python. That worked fine because there was never any issue with the GIL being released or acquired and the Python VM could be guaranteed to be in a consistent state. For multi-threaded operation, we cannot really guarantee that we will have the GIL when we need it without a great deal of hassle and a performance hit. Bad things will happen if C++ code attempts to callback into Python when the GIL is not acquired.
Here is a (not necessarily exhaustive) list of alternative mechanisms for user notification: () Use C++ output streams. (We shouldn't have to worry about interference with Python streams since both are not in use at the same time. And, the way the reporting is structured is such that we shouldn't have to worry about multiple threads clobbering each other's output either. I'm +1 on this solution unless I think of something better.) () Redesign the code to perform a certain amount of work, return to the Python wrapper for reporting, and then be resumed. (Potentially messy and would almost certainly break some existing interfaces. I'm -1 on this.) () Setup a dedicated listener thread and send messages to it via an *intra-process communication mechanism of some sort. (Would be a fun excuse to learn about ZeroMQ's inproc stuff, but probably not worth the hassle. So, I'm -1 on this as well.) (*) Set up a synchronization barrier among the threads for getting the lock, reporting, and releasing the lock. (This will hurt performance, possibly quite badly. It will also ruin the threading model-agnostic scheme we currently have. I'm definitely -1 on this.)
Other ideas welcome.
Yeah, all those sound grim. My goal is to allow flexibility, but not necessarily for the production scripts (where C++ output streams would be a good solution). Would something like #2 (periodic checking in) have much overhead in scripts where it was never needed? Perhaps we could introduce a general callback mechanism in C++ and enable Python to make use of it, but not have it a "standard" feature in our production scripts?
--titus
C. Titus Brown, ctb@msu.edu
emcd
mr-c
We have various places where we wish to notify the user of progress being made during processing. In the original single-threaded model, callbacks were made from the C++ code doing the heavy-lifting to Python. That worked fine because there was never any issue with the GIL being released or acquired and the Python VM could be guaranteed to be in a consistent state. For multi-threaded operation, we cannot really guarantee that we will have the GIL when we need it without a great deal of hassle and a performance hit. Bad things will happen if C++ code attempts to callback into Python when the GIL is not acquired.
Here is a (not necessarily exhaustive) list of alternative mechanisms for user notification: () Use C++ output streams. (We shouldn't have to worry about interference with Python streams since both are not in use at the same time. And, the way the reporting is structured is such that we shouldn't have to worry about multiple threads clobbering each other's output either. I'm +1 on this solution unless I think of something better.) () Redesign the code to perform a certain amount of work, return to the Python wrapper for reporting, and then be resumed. (Potentially messy and would almost certainly break some existing interfaces. I'm -1 on this.) () Setup a dedicated listener thread and send messages to it via an *intra-process communication mechanism of some sort. (Would be a fun excuse to learn about ZeroMQ's inproc stuff, but probably not worth the hassle. So, I'm -1 on this as well.) (*) Set up a synchronization barrier among the threads for getting the lock, reporting, and releasing the lock. (This will hurt performance, possibly quite badly. It will also ruin the threading model-agnostic scheme we currently have. I'm definitely -1 on this.)
Other ideas welcome.