Locks in the standard library should be sanitized on fork

[gpshead]

BPO	6721
Nosy	@rhettinger, @gpshead, @vsajip, @jcea, @nirs, @pitrou, @vstinner, @applio, @cagney, @Birne94, @ochedru, @kevans91, @jessefarnham, @rojer, @koubaa
PRs	python/cpython#4071 python/cpython#9291 python/cpython#21986 python/cpython#22205 python/cpython#22651
Files	lock_fork_thread_deadlock_demo.py forklocktests.patch reinit_locks.diff: patch adding locks reinitialization upon fork emit_warning_on_fork.patch atfork.patch reinit_locks_2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.7', 'type-feature', 'library'] title = 'Locks in the standard library should be sanitized on fork' updated_at = user = 'https://github.com/gpshead' ``` bugs.python.org fields: ```python activity = actor = 'kevans' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'gregory.p.smith' dependencies = [] files = ['14740', '21874', '22005', '22525', '24303', '25776'] hgrepos = [] issue_num = 6721 keywords = ['patch'] message_count = 133.0 messages = ['91674', '91936', '92766', '94102', '94115', '94133', '94135', '128282', '128307', '128311', '128316', '128369', '135012', '135067', '135069', '135079', '135083', '135095', '135096', '135143', '135157', '135173', '135543', '135857', '135866', '135897', '135899', '135948', '135965', '135984', '136003', '136039', '136045', '136047', '136120', '136147', '139084', '139245', '139470', '139474', '139480', '139485', '139488', '139489', '139509', '139511', '139521', '139522', '139584', '139599', '139608', '139800', '139808', '139850', '139852', '139858', '139869', '139897', '139929', '140215', '140402', '140550', '140658', '140659', '140668', '140689', '140690', '140691', '141286', '143174', '143274', '143279', '151168', '151266', '151267', '151845', '151846', '151853', '161019', '161029', '161389', '161405', '161470', '161953', '162019', '162031', '162034', '162036', '162038', '162039', '162040', '162041', '162053', '162054', '162063', '162113', '162114', '162115', '162117', '162120', '162137', '162160', '270015', '270017', '270018', '270019', '270020', '270021', '270022', '270023', '270028', '289716', '294726', '294834', '304714', '304716', '304722', '304723', '314983', '325326', '327267', '329474', '339369', '339371', '339393', '339418', '339454', '339458', '339473', '365169', '367528', '367702', '368882'] nosy_count = 29.0 nosy_names = ['rhettinger', 'gregory.p.smith', 'vinay.sajip', 'jcea', 'nirs', 'pitrou', 'vstinner', 'nirai', 'forest_atq', 'ionelmc', 'bobbyi', 'neologix', 'Giovanni.Bajo', 'sdaoden', 'tshepang', 'sbt', 'lesha', 'dan.oreilly', 'davin', 'Connor.Wolf', 'Winterflower', 'cagney', 'Birne94', 'ochedru', 'kevans', 'jesse.farnham', 'hugh', 'rojer', 'koubaa'] pr_nums = ['4071', '9291', '21986', '22205', '22651'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue6721' versions = ['Python 3.7'] ```

[gpshead]

The python logging module uses a lock to surround many operations, in particular. This causes deadlocks in programs that use logging, fork and threading simultaneously.

1) spawn one or more threads in your program 2) have at least one of those threads make logging calls that will be emitted. 3) have your main thread or another thread use os.fork() to run some python code in a child process. 4) If the fork happened while one of your threads was within the logging.Handler.handle() critical section (or anywhere else where handler.lock is acquired), your child process will deadlock as soon as it tries to log anything. It inherited a held lock.

The deadlock is more likely to happen on a highly loaded system which tends to widen the deadlock opportunity window due to context switching.

A demo of the problem simplified into one file is attached.

The Python standard library should not be the cause of these deadlocks.
We need a way for all standard library locks to be cleaned up when forking. By doing one of the following:

A) acquire all locks before forking, release them immediately after. B) forceably release all standard library locks after forking in the child process.

Code was added to call some cleanups after forking in http://bugs.python.org/issue874900 but there are more things that also need this same sort of cleanup (logging for example).

Rather than having to manually add after fork code hooks into every file in the standard library that uses locks, a more general solution to track and manage locks across fork would be a good idea.

[gpshead]

I've started a project to patch this and similar messes up for Python 2.4 and later here:

http://code.google.com/p/python-atfork/

I'd like to take ideas or implementations from that when possible for future use in the python standard library.

[gpshead]

bpo-6923 has been opened to provide a C API for an atfork mechanism for use by extension modules.

[pitrou]

Rather than having a kind of global module registry, locks could keep track of what was the last PID, and reinitialize themselves if it changed. This is assuming getpid() is fast :-)

[gpshead]

Antoine Pitrou \pitrou@free.fr\ added the comment:

Rather than having a kind of global module registry, locks could keep track of what was the last PID, and reinitialize themselves if it changed. This is assuming getpid() is fast :-)

Locks can't blindly release themselves because they find themselves running in another process.

If anything if a lock is held and finds itself running in a new process any attempt to use the lock should raise an exception so that the bug is noticed.

I'm not sure a PID check is good enough. old linux using linuxthreads had a different pid for every thread, current linux with NPTL is more like other oses with the same pid for all threads.

[pitrou]

I was suggesting "reinitialize", rather than "release". That is, create a new lock (mutex, semaphore, etc.) and let the old one die (or occupy some tiny bit of memory).

[gpshead]

no need for that. the problem is that they're held by a thread that does not exist in the newly forked child process so they will never be released in the new process.

example: if you fork while another thread is in the middle of logging something and then try to log something yourself in the child, your child process will deadlock on the logging module's lock.

locks are not shared between processes so reinitializing them with a new object wouldn't do anything.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

I'm not sure that releasing the mutex is enough, it can still lead to a segfault, as is probably the case in this issue : http://bugs.python.org/issue11148

Quoting pthread_atfork man page :

To understand the purpose of pthread_atfork, recall that fork duplicates the whole memory space, including mutexes in their current locking state, but only the calling thread: other threads are not running in the child process. The mutexes are not usable after the fork and must be initialized with pthread_mutex_init in the child process. This is a limitation of the current implementation and might or might not be present in future versions.

To avoid this, install handlers with pthread_atfork as follows: have the prepare handler lock the mutexes (in locking order), and the parent handler unlock the mutexes. The child handler should reset the mutexes using pthread_mutex_init, as well as any other synchronization objects such as condition variables.

Locking the global mutexes before the fork ensures that all other threads are locked out of the critical regions of code protected by those mutexes. Thus when fork takes a snapshot of the parent's address space, that snapshot will copy valid, stable data. Resetting the synchronization objects in the child process will ensure they are properly cleansed of any artifacts from the threading subsystem of the parent process. For example, a mutex may inherit a wait queue of threads waiting for the lock; this wait queue makes no sense in the child process. Initializing the mutex takes care of this.

pthread_atfork might be worth looking into

[gpshead]

fwiw http://bugs.python.org/issue6643 recently fixed on issue where a mutex was being closed instead of reinitialized after a fork. more are likely needed.

Are you suggesting to use pthread_atfork to call pthread_mutex_init on all mutexes created by Python in the child after a fork? I'll have to ponder that some more. Given the mutexes are all useless post fork it does not strike me as a bad idea.

[pitrou]

Are you suggesting to use pthread_atfork to call pthread_mutex_init on > all mutexes created by Python in the child after a fork? I'll have to > ponder that some more. Given the mutexes are all useless post fork it > does not strike me as a bad idea.

I don't really understand. It's quite similar to the idea you shot down in msg94135. Or am I missing something?

[gpshead]

Yeah, I'm trying to figure out what I was thinking then or if I was just plain wrong. :)

I was clearly wrong about a release being done in the child being the right thing to do (bpo-6643 proved that, the state held by a lock is not usable to another process on all platforms such that release even works).

Part of it looks like I wanted a way to detect it was happening as any lock that is held during a fork indicates a _potential_ bug (the lock wasn't registered anywhere to be released before the fork) but not everything needs to care about that.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

I was clearly wrong about a release being done in the child being the right thing to do (bpo-6643 proved that, the state held by a lock is not usable to another process on all platforms such that release even works).

Yeah, apparently OS-X is one of them, the reporter in bpo-11148 is experiencing segfaults under OS-X.

Are you suggesting to use pthread_atfork to call pthread_mutex_init on all mutexes created by Python in the child after a fork? I'll have to ponder that some more. Given the mutexes are all useless post fork it does not strike me as a bad idea.

Yes, that's what I was thinking. Instead of scattering the lock-reclaiming code all over the place, try to use a more standard API specifically designed with that in mind. Note the base issue is that we're authorizing things which are forbidden : in a multi-threaded process, only sync-safe calls are authorized between fork and exec.

2011/2/10 Gregory P. Smith \report@bugs.python.org\:

Gregory P. Smith \greg@krypto.org\ added the comment:

Yeah, I'm trying to figure out what I was thinking then or if I was just plain wrong. :)

I was clearly wrong about a release being done in the child being the right thing to do (bpo-6643 proved that, the state held by a lock is not usable to another process on all platforms such that release even works).

Part of it looks like I wanted a way to detect it was happening as any lock that is held during a fork indicates a _potential_ bug (the lock wasn't registered anywhere to be released before the fork) but not everything needs to care about that.

---------- versions: +Python 3.3

---

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue6721\

---

[pitrou]

I encountered this issue while debugging some multiprocessing code; fork() would be called from one thread while sys.stdout was in use in another thread (simply because of a couple of debugging statements). As a result the IO lock would be already "taken" in the child process and any operation on sys.stdout would deadlock.

This is definitely something that can happen more easily than I thought.

[pitrou]

Here is a patch with tests for the issue (some of which fail of course). Do we agree that these tests are right?

[gpshead]

Those tests make sense to me.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

# A lock taken from the current thread should stay taken in the # child process.

Note that I'm not sure of how to implement this. After a fork, even releasing the lock can be unsafe, it must be re-initialized, see following comment in glibc's malloc implementation: / In NPTL, unlocking a mutex in the child process after a fork() is currently unsafe, whereas re-initializing it is safe and does not leak resources. Therefore, a special atfork handler is installed for the child. \/

Note that this means that even the current code allocating new locks after fork (in Lib/threading.py, _after_fork and _reset_internal_locks) is unsafe, because the old locks will be deallocated, and the lock deallocation tries to acquire and release the lock before destroying it (in issue bpo-11148 the OP experienced a segfault on OS-X when locking a mutex, but I'm not sure of the exact context).

Also, this would imply keeping track of the thread currently owning the lock, and doesn't match the typical pthread_atfork idiom (acquire locks just before fork, release just after in parent and child, or just reinit them in the child process)

Finally, IMHO, forking while holding a lock and expecting it to be usable after fork doesn't make much sense, since a lock is acquired by a thread, and this threads doesn't exist in the child process. It's explicitely described as "undefined" by POSIX, see http://pubs.opengroup.org/onlinepubs/007908799/xsh/sem_init.html : """ The use of the semaphore by threads other than those created in the same process is undefined. """

So I'm not sure whether it's feasable/wise to provide such a guarantee.

[pitrou]

Also, this would imply keeping track of the thread currently owning the lock,

Yes, we would need to keep track of the thread id and process id inside the lock. We also need a global variable of the main thread id after fork, and a per-lock "taken" flag.

Synopsis:

    def _reinit_if_needed(self):
        # Call this before each acquire() or release()
        if self.pid != getpid():
            sem_init(self.sem, 0, 1)
            if self.taken:
                if self.tid == main_thread_id_after_fork:
                    # Lock was taken in forked thread, re-take it
                    sem_wait(self.sem)
                else:
                    # It's now released
                    self.taken = False
            self.pid = getpid()
            self.tid = current_thread_id()

and doesn't match the typical pthread_atfork idiom (acquire locks just before fork, release just after in parent and child, or just reinit them in the child process)

Well, I fail to understand how that idiom can help us. We're not a self-contained application, we're a whole programming language. Calling fork() only when no lock is held is unworkable (for example, we use locks around buffered I/O objects).

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Yes, we would need to keep track of the thread id and process id inside the lock. We also need a global variable of the main thread id after fork, and a per-lock "taken" flag.

Synopsis:

   def _reinit_if_needed(self):        # Call this before each acquire() or release()        if self.pid != getpid():            sem_init(self.sem, 0, 1)            if self.taken:                if self.tid == main_thread_id_after_fork:                    # Lock was taken in forked thread, re-take it                    sem_wait(self.sem)                else:                    # It's now released                    self.taken = False            self.pid = getpid()            self.tid = current_thread_id()

A couple remarks:

• with linuxthreads, different threads within the same process have the same PID - it may be true for other implementations - so this would lead to spurious reinitializations
• what's current_thread_id ? If it's thread_get_ident (pthread_self), since TID is not guaranteed to be inherited across fork, this won't work
• calling getpid at every acquire/release is expensive, even though it's a trivial syscall (it'll have to measured though)
• imagine the following happens:

P1

lock.acquire()
fork()    ->       P2
                       start_new_thread T2
                       T1           T2
                                        lock.acquire()

The acquisition of lock by T2 will cause lock's reinitialization: what happens to the lock wait queue ? who owns the lock ? That why I don't think we can delay the reinitialization of locks, but I could be wrong.

Well, I fail to understand how that idiom can help us. We're not a self-contained application, we're a whole programming language. Calling fork() only when no lock is held is unworkable (for example, we use locks around buffered I/O objects).

Yes, but in that case, you don't have to reacquire the locks after fork. In the deadlock you experienced above, the thread that forked wasn't the one in the I/O code, so the corresponding lock can be re-initialized anyway, since the thread in the I/O code at that time won't exist after fork. And it's true with every lock in the library code: they're only held in short critical sections (typically acquired when entering a function and released when leaving), and since it's not the threads inside those libraries that fork, all those locks can simply be reinitialized on fork, without having the reacquire them.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Oops, for liunxthreads, you should of course read "different PIDs", not "same PID".

[pitrou]

• what's current_thread_id ? If it's thread_get_ident (pthread_self), since TID is not guaranteed to be inherited across fork, this won't work

Ouch, then the approach I'm proposing is probably doomed.

And it's true with every lock in the library code: they're only held in short critical sections (typically acquired when entering a function and released when leaving), and since it's not the threads inside those libraries that fork, all those locks can simply be reinitialized on fork, without having the reacquire them.

Well, this means indeed that *some* locks can be handled, but not all of them and not automatically, right? Also, how would you propose they be dealt with in practice? How do they get registered, and how does the reinitialization happen?

(do note that library code can call arbitrary third-party code, by the way: for example through encodings in the text I/O layer)

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

> - what's current_thread_id ? If it's thread_get_ident (pthread_self), > since TID is not guaranteed to be inherited across fork, this won't > work

Ouch, then the approach I'm proposing is probably doomed.

Well, it works on Linux with NPTL, but I'm not sure at all it holds for other implementations (pthread_t it's only meaningful within the same process). But I'm not sure it's really the "killer" point: PID with linuxthreads and lock being acquired by a second thread before the main thread releases it in the child process also look like serious problems.

Well, this means indeed that *some* locks can be handled, but not all of them and not automatically, right? Also, how would you propose they be dealt with in practice? How do they get registered, and how does the reinitialization happen?

When a lock object is allocated in Modules/threadmodule.c (PyThread_allocate_lock/newlockobject), add the underlying lock (self->lock_lock) to a linked list (since it's called with the GIL held, we don't need to protect the linked list from concurrent access). Each thread implementation (thread_pthread.h, thread_nt.h) would provide a new PyThread_reinit_lock function that would do the right thing (pthread_mutex_destroy/init, sem_destroy/init, etc). Modules/threadmodule.c would provide a new PyThread_ReInitLocks that would walk through the linked list and call PyThread_reinit_lock for each lock. PyOS_AfterFork would call this PyThread_ReInitLocks right after fork. This would have the advantage of being consistent with what's already done to reinit the TLS key and the import lock. So, we guarantee to be in a consistent and usable state when PyOS_AfterFork returns. Also, it's somewhat simpler because we're sure that at that point only one thread is running (once again, no need to protect the linked-list walk). I don't think that the performance impact would be noticable (I know it's O(N) where N is the number of locks), and contrarily to the automatic approach, this wouldn't penalize every acquire/release. Of course, this would solve the problem of threading's module locks, so PyEval_ReInitThreads could be removed, along with threading.py's _after_fork and _reset_internal_locks. In short, this would reset every lock held so that they're usable in the child process, even locks allocated e.g. from Modules/_io/bufferedio.c. But this wouldn't allow a lock's state to be inherited across fork for the main thread (but like I said, I don't think that this makes much sense anyway, and to my knowledge no implementation makes such a guarantee - and definitely not POSIX).

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Please disregard my comment on PyEval_ReInitThreads and _after_fork: it will of course still be necessary, because it does much more than just reinitializing locks (e.g. stop threads). Also, note that both approaches don't handle synchronization primitives other than bare Lock and RLock. For example, Condition and Event used in the threading module wouldn't be reset automatically: that's maybe something that could be handled by Gregory's atfork mechanism.

[pitrou]

Thanks for the explanations. This sounds like an interesting path.

Each thread implementation (thread_pthread.h, thread_nt.h) would provide a new PyThread_reinit_lock function that would do the right thing (pthread_mutex_destroy/init, sem_destroy/init, etc). Modules/threadmodule.c would provide a new PyThread_ReInitLocks that would walk through the linked list and call PyThread_reinit_lock for each lock.

Actually, I think the issue is POSIX-specific: Windows has no fork(), and we don't care about other platforms anymore (they are, are being, or will be soon deprecated). It means only the POSIX implementation needs to register its locks in a linked list.

But this wouldn't allow a lock's state to be inherited across fork for the main thread (but like I said, I don't think that this makes much sense anyway, and to my knowledge no implementation makes such a guarantee - and definitely not POSIX).

Well, the big difference between Python locks and POSIX mutexes is that Python locks can be released from another thread. They're a kind of trivial semaphore really, and this makes them usable for other purpose than mutual exclusion (you can e.g. use a lock as a simple event by blocking on a second acquire() until another thread calls release()).

But even though we might not be "fixing" arbitrary Python code automatically, fixing the interpreter's internal locks (especially the IO locks) would be great already.

(we could also imagine that the creator of the lock decides whether it should get reinitialized after fork)

[7b124e7b-b61d-49f6-9762-fd36b627e237]

Hi,

There seem to be two alternatives for atfork handlers: 1) acquire locks during prepare phase and unlock them in parent and child after fork. 2) reset library to some consistent state in child after fork.

http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html

Option (2) makes sense but is probably not always applicable. Option (1) depends on being able to acquire locks in locking order, but how can we determine correct locking order across libraries?

Initializing locks in child after fork without acquiring them before the fork may result in corrupted program state and so is probably not a good idea.

On a positive note, if I understand correctly, Python signal handler functions are actually run in the regular interpreter loop (as pending calls) after the signal has been handled and so os.fork() atfork handlers will not be restricted to async-signal-safe operations (since a Python fork is never done in a signal handler).

http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html "It is therefore undefined for the fork handlers to execute functions that are not async-signal-safe when fork() is called from a signal handler."

Opinion by Butenhof who was involved in the standardization effort of POSIX threads: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde

...so how can we establish correct (cross library) locking order during prepare stage?

Nir

[5792609d-7136-4bf5-a72c-931da2480f6a]

@Nir Aides: *thanks* for this link: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde You made my day!

[pitrou]

...so how can we establish correct (cross library) locking order during prepare stage?

That sounds like a lost battle, if it requires the libraries' cooperation. I think resetting locks is the best we can do. It might not work ok in all cases, but if it can handle simple cases (such as I/O and logging locks), it is already very good.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Hi,

Hello Nir,

Option (2) makes sense but is probably not always applicable. Option (1) depends on being able to acquire locks in locking order, but how can we determine correct locking order across libraries?

There are indeed a couple problems with 1: 1) actually, releasing the mutex/semaphore from the child is not guaranteed to be safe, see this comment from glibc's malloc: / In NPTL, unlocking a mutex in the child process after a fork() is currently unsafe, whereas re-initializing it is safe and does not leak resources. Therefore, a special atfork handler is installed for the child. \/ We could just destroy/reinit them, though.

2) acquiring locks just before fork is probably one of the best way to deadlock (acquiring a lock we already hold, or acquiring a lock needed by another thread before it releases its own lock). Apart from adding dealock avoidance/recovery mechanisms - which would be far from trivial - I don't see how we could solve this, given that each library can use its own locks, not counting the user-created ones

3) there's another special lock we must take into account, the GIL: contrarily to a typical C program, we can't have the thread forking blindly try to acquire all locks just before fork, because since we hold the GIL, other threads won't be able to proceed (unless of course they're in a section where they don't run without the GIL held).

So, we would have to:

• release the GIL
• acquire all locks in the correct order
• re-acquire the GIL
• fork
• reinit all locks after fork

I think this is going to be very complicated.

4) Python locks differ from usual mutexes/semaphores in that they can be held for quite some time (for example while performing I/O). Thus, acquiring all the locks could take a long time, and users might get irritated if fork takes 2 seconds to complete.

5) Finally, there's a fundamental problem with this approach, because Python locks can be released by a thread other than the one that owns it. Imagine this happens:

T1 T2 lock.acquire() (do something without releasing lock) fork() lock.release()

This is perfectly valid with the current lock implementation (for example, it can be used to implement a rendez-vous point so that T2 doesn't start processing before T1 forked worker processes, or whatever). But if T1 tries to acquire lock (held by T2) before fork, then it will deadlock, since it will never be release by T2.

For all those reasons, I don't think that this approach is reasonable, but I could be wrong :-)

Initializing locks in child after fork without acquiring them before the fork may result in corrupted program state and so is probably not a good idea.

Yes, but in practise, I think that this shouldn't be too much of a problem. Also note that you can very well have the same type of problem with sections not protected explicitely by locks: for example, if you have a thread working exclusively on an object (maybe part of a threadpool), a fork can very well happen while the object is in an inconsistent state. Acquiring locks before fork won't help that. But I think this should eventually be addressed, maybe by specific atfork handlers.

On a positive note, if I understand correctly, Python signal handler functions are actually run in the regular interpreter loop (as pending calls) after the signal has been handled and so os.fork() atfork handlers will not be restricted to async-signal-safe operations (since a Python fork is never done in a signal handler).

That's correct.

In short, I think that we could first try to avoid common deadlocks by just resetting locks in the child process. This is not panacea, but this should solve the vast majority of deadlocks, and would open the door to potential future refinements using atfork-like handlers.

Attached is a first draft for a such patch (with tests). Synopsis:

• when a PyThread_type_lock is created, it's added to a linked-list, when it's deleted, it's removed from the linked list
• PyOS_AfterFork() calls PyThread_ReinitLocks() which calls PyThread_reinit_lock() for each lock in the linked list
• PyThread_reinit_lock() does the right thing (i.e. sem_destroy/init for USESEMAPHORES and pthread(mutex|cond)_destroy/init for emulated semaphores).

Notes:

• since it's only applicable to POSIX (since other Unix thread implementations will be dropped), I've only defined a PyThread_ReinitLocks inside Python/thread_pthread.h, so it won't build on other platforms. How should I proceed: like PyThread_ReInitTLS(), add a stub function to all Python/thread_xxx.h, or guard the call to PyThread_ReinitLocks() with #ifdef _POSIX_THREADS ?
• I'm not sure of how to handle sem_init/etc failures in the reinit code: for now I just ignore this possibility, like what's done for the import lock reset
• insertions/removals from the linked list are not protected from concurrent access because I assume that locks are created/deleted with the GIL held: is that a reasonable assumption, or should I add a mutex to protect those accesses?

This fixes common deadlocks with threading.Lock, and PyThread_type_lock (used for example by I/O code).

[5792609d-7136-4bf5-a72c-931da2480f6a]

@ Charles-François Natali \report@bugs.python.org\ wrote (2011-05-13 13:24+0200):

I happily posted a reinit patch

I must say in advance that we have implemented our own thread support 2003-2005 and i'm thus lucky not to need to use anything else ever since. So. And of course i have no overview about Python. But i looked and saw no errors in the default path and the tests run without errors. Then i started to try your semaphore path which is a bit problematic because Mac OS X doesn't offer anon sems ;). ( By the way, in PyThread_acquire_lock_timed() these lines

if (microseconds > 0)
            MICROSECONDS_TO_TIMESPEC(microseconds, ts);

result in these compiler warnings.

python/thread_pthread.h: In function ‘PyThread_acquire_lock_timed’: Python/thread_pthread.h:424: warning: ‘ts.tv_sec’ may be used uninitialized in this function Python/thread_pthread.h:424: warning: ‘ts.tv_nsec’ may be used uninitialized in this function )

#ifdef USE_SEMAPHORES
#define broken_sem_init broken_sem_init
static int broken_sem_init(sem_t **sem, int shared, unsigned int value) {
    int ret;
    auto char buffer[32];
    static long counter = 3000;
    sprintf(buffer, "%016ld", ++counter);
    *sem = sem_open(buffer, O_CREAT, (mode_t)0600, (unsigned int)value);
    ret = (*sem == SEM_FAILED) ? -1 : 0;
    //printf("BROKEN_SEM_INIT WILL RETURN %d (value=%u)\n", ret,value);
    return ret;
}
static int sem_timedwait(sem_t *sem, struct timespec *ts) {
    int success = -1, iters = 1000;
    struct timespec now, wait;
    printf("STARTING LOOP\n");
    for (;;) {
        if (sem_trywait(sem) == 0) {
            printf("TRYWAIT OK\n");
            success = 0;
            break;
        }
        wait.tv_sec = 0, wait.tv_nsec = 200 * 1000;
        //printf("DOWN "); fflush(stdout);
        nanosleep(&wait, NULL);
        MICROSECONDS_TO_TIMESPEC(0, now);
        //printf("WOKE UP NOW=%ld:%ld END=%ld:%ld\n", now.tv_sec,now.tv_nsec, ts->tv_sec,ts->tv_nsec);
        if (now.tv_sec > ts->tv_sec ||
            (now.tv_sec == ts->tv_sec && now.tv_nsec >= ts->tv_nsec))
            break;
        if (--iters < 0) {
            printf("BREAKING OFF LOOP, 1000 iterations\n");
            errno = ETIMEDOUT;
            break;
        }
    }
    return success;
}
#define sem_destroy sem_close

typedef struct _pthread_lock {
    sem_t               *sem;
    struct _pthread_lock*next;
    sem_t               sem_buf;
} pthread_lock;
#endif

plus all the changes the struct change implies, say. Yes it's silly, but i wanted to test. And this is the result:

== CPython 3.3a0 (default:804abc2c60de+, May 14 2011, 01:09:53) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] == Darwin-10.7.0-i386-64bit little-endian == /Users/steffen/src/cpython/build/test_python_19230 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=0, quiet=0) Using random seed 1362049 [1/1] test_threading STARTING LOOP test_acquire_contended (test.test_threading.LockTests) ... ok test_acquire_destroy (test.test_threading.LockTests) ... ok test_acquire_release (test.test_threading.LockTests) ... ok test_constructor (test.test_threading.LockTests) ... ok test_different_thread (test.test_threading.LockTests) ... ok test_reacquire (test.test_threading.LockTests) ... ok test_state_after_timeout (test.test_threading.LockTests) ... ok test_thread_leak (test.test_threading.LockTests) ... ok test_timeout (test.test_threading.LockTests) ... STARTING LOOP TRYWAIT OK FAIL test_try_acquire (test.test_threading.LockTests) ... ok test_try_acquire_contended (test.test_threading.LockTests) ... ok test_with (test.test_threading.LockTests) ... ok testis_owned (test.test_threading.PyRLockTests) ... ok test_acquire_contended (test.test_threading.PyRLockTests) ... ok test_acquire_destroy (test.test_threading.PyRLockTests) ... ok test_acquire_release (test.test_threading.PyRLockTests) ... ok test_constructor (test.test_threading.PyRLockTests) ... ok test_different_thread (test.test_threading.PyRLockTests) ... ok test_reacquire (test.test_threading.PyRLockTests) ... ok test_release_unacquired (test.test_threading.PyRLockTests) ... ok test_thread_leak (test.test_threading.PyRLockTests) ... ok test_timeout (test.test_threading.PyRLockTests) ... STARTING LOOP TRYWAIT OK FAIL test_try_acquire (test.test_threading.PyRLockTests) ... ok test_try_acquire_contended (test.test_threading.PyRLockTests) ... ok test_with (test.test_threading.PyRLockTests) ... ok test__is_owned (test.test_threading.CRLockTests) ... ok test_acquire_contended (test.test_threading.CRLockTests) ... ok test_acquire_destroy (test.test_threading.CRLockTests) ... ok test_acquire_release (test.test_threading.CRLockTests) ... ok test_constructor (test.test_threading.CRLockTests) ... ok test_different_thread (test.test_threading.CRLockTests) ... ok test_reacquire (test.test_threading.CRLockTests) ... ok test_release_unacquired (test.test_threading.CRLockTests) ... ok test_thread_leak (test.test_threading.CRLockTests) ... BREAKING OFF LOOP, 1000 iterations Timeout (1:00:00)! Thread 0x00007fff70677ca0: File "/Users/steffen/src/cpython/Lib/test/lock_tests.py", line 17 in _wait File "/Users/steffen/src/cpython/Lib/test/lock_tests.py", line 52 in wait_for_finished File "/Users/steffen/src/cpython/Lib/test/lock_tests.py", line 152 in test_threadleak File "/Users/steffen/src/cpython/Lib/unittest/case.py", line 407 in _executeTestPart File "/Users/steffen/src/cpython/Lib/unittest/case.py", line 462 in run File "/Users/steffen/src/cpython/Lib/unittest/case.py", line 514 in \_call File "/Users/steffen/src/cpython/Lib/unittest/suite.py", line 105 in run File "/Users/steffen/src/cpython/Lib/unittest/suite.py", line 67 in __call File "/Users/steffen/src/cpython/Lib/unittest/suite.py", line 105 in run File "/Users/steffen/src/cpython/Lib/unittest/suite.py", line 67 in __call File "/Users/steffen/src/cpython/Lib/unittest/runner.py", line 168 in run File "/Users/steffen/src/cpython/Lib/test/support.py", line 1187 in _run_suite File "/Users/steffen/src/cpython/Lib/test/support.py", line 1213 in run_unittest File "/Users/steffen/src/cpython/Lib/test/test_threading.py", line 748 in test_main File "/Users/steffen/src/cpython/Lib/test/regrtest.py", line 1044 in runtest_inner File "/Users/steffen/src/cpython/Lib/test/regrtest.py", line 838 in runtest File "/Users/steffen/src/cpython/Lib/test/regrtest.py", line 662 in main File "/Users/steffen/src/cpython/Lib/test/main.py", line 13 in \<module> File "/Users/steffen/src/cpython/Lib/runpy.py", line 73 in _run_code File "/Users/steffen/src/cpython/Lib/runpy.py", line 160 in _run_module_as_main

Hope that helps a bit. Bâillement.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Hello Steffen,

First, thanks for testing this on OS-X: I only have access to Linux systems (I tested both the semaphore and the emulated semaphore paths).

If I understand correctly, the patch works fine with the default build option on OS-X. Then, you're saying that OS-X doesn't have POSIX unnamed semaphores: this means that the default build uses the mutex+condition variable version. Am I correct? But I don't understand the last part of your message. Do you mean that you implemented your own version of the semaphore path using named semaphores, and that it fails with my patch (well, your adapted version of it) ? If yes, then you'll understand that I can't comment on this, since it's not my code :-) But after a quick look at the code you posted, I think that your acquire code is broken: sem_timedwait(timeout) if not the same as trying sem_trywait multiple times until timeout expires: in case of contention, this will fail.

[7b124e7b-b61d-49f6-9762-fd36b627e237]

I think that generally it is better to deadlock than corrupt data.

2) acquiring locks just before fork is probably one of the best way to deadlock (acquiring a lock we already hold, or acquiring a lock needed by another thread before it releases its own lock). Apart from adding dealock avoidance/recovery mechanisms - which would be far from trivial - I don't see how we could solve this, given that each library can use its own locks, not counting the user-created ones

a) We know the correct locking order in Python's std libraries so the problem there is kind of solved.

b) We can put the burden of other locks on application developers and since currently no one registers atfork handlers, there is no problem there yet.

4) Python locks differ from usual mutexes/semaphores in that they can be held for quite some time (for example while performing I/O). Thus, acquiring all the locks could take a long time, and users might get irritated if fork takes 2 seconds to complete.

We only need a prepare handler to acquire locks that protect data from corruption.

A lock synchronizing IO which is held for long periods may possibly be initialized in child without being acquired in a prepare handler; for example, a lock serializing logging messages.

In other cases or in general an atfork handler may reset or reinitialize a library without acquiring locks in a prepare handler.

5) Finally, there's a fundamental problem with this approach, because Python locks can be released by a thread other than the one that owns it. Imagine this happens:

T1 T2 lock.acquire() (do something without releasing lock) fork() lock.release()

This is perfectly valid with the current lock implementation (for example, it can be used to implement a rendez-vous point so that T2 doesn't start processing before T1 forked worker processes, or whatever). But if T1 tries to acquire lock (held by T2) before fork, then it will deadlock, since it will never be release by T2.

I think we do not need to acquire rendezvous locks in a prepare handler.

> Initializing locks in child after fork without acquiring them before the > fork may result in corrupted program state and so is probably not a good > idea.

Yes, but in practise, I think that this shouldn't be too much of a problem. Also note that you can very well have the same type of problem with sections not protected explicitely by locks: for example, if you have a thread working exclusively on an object (maybe part of a threadpool), a fork can very well happen while the object is in an inconsistent state. Acquiring locks before fork won't help that.

I think a worker thread that works exclusively on an object does not create the problem: a) If the fork thread eventually needs to read the object then you need synchronization. b) If the worker thread eventually writes data into file or DB then that operation will be completed at the parent process.

To summarize I think we should take the atfork path. An atfork handler does not need to acquire all locks, but only those required by library logic, which the handler is aware of, and as a bonus it can be used to do all sort of stuff such as cleaning up, reinitializing a library, etc...

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

a) We know the correct locking order in Python's std libraries so the problem there is kind of solved.

I think that you're greatly under-estimating the complexity of lock ordering. If we were just implementing a malloc implementation protected with a single mutex, then yes, it would be simple. But here, you have multiple libraries with each their own locks, locks at the I/O layer, in the socket module (some name resolution libraries are not thread-safe), and in many other places. And all those interact. For example, buffered I/O objects each have their own lock (Antoine, correct me if I'm wrong). It's a common cause of deadlock. Now imagine I have a thread that logs information to a bz2 stream, so that it's compressed on-the-fly. Sounds reasonable, no? Well, the lock hierarchy is:

buffered stream lock bz2-level lock logging object I/O lock

Do you still think that getting the locking order right is easy?

Another example, with I/O locks (and if you're concerned with data corruption, those are definitely the one you would want to handle with atfork): I have a thread blocking on a write (maybe the output pipe is full, maybe it's a NFS file system and the server takes a long time to respond, etc. Or maybe it's just waiting for someone to type something on stdin.). Another thread forks. The atfork-handler will try to acquire the buffered I/O object's lock: it won't succeed until the other threads finally manages to write/read. It could take seconds, or forever. And there are many other things that could go wrong, because contrarily to a standalone and self-contained library, Python is made of several components, at different layers, that can call each other in an arbitrary order. Also, some locks can be held for arbitrarily long.

That's why I still think that this can be fully handled by atfork handlers.

But don't get me wrong: like you, I think that we should definitely have an atfork mechanism. I just think it won't be able to solve all the issues, and that I can also bring its own set of troubles.

Concerning the risk of corruption (invariant broken), you're right. But resetting the locks is the approach currently in use for the threading module, and it seems to work reasonably well there.

Finally, I'd just like to insist on a point: In a multi-threaded program, between fork and exec, the code must be async-safe. This means that in theory, you can't call pthread_mutex_release/pthread_mutex_destroy, fwrite, malloc, etc. Period. This means that in theory, we shouldn't be running Python code at all! So if we really wanted to be safe, the only solution would be to forbid fork() in a multi-threaded program. Since it's not really a reasonable option, and that the underlying platform (POSIX) doesn't allow to be safe either, I guess that the only choice left is to provide a bet-try implementation, knowing perfectly that there will always be some corner cases that can't be handled.

[5792609d-7136-4bf5-a72c-931da2480f6a]

@ Charles-François Natali wrote (2011-05-15 01:14+0200):

So if we really wanted to be safe, the only solution would be to forbid fork() in a multi-threaded program. Since it's not really a reasonable option

But now - why this? The only really acceptable thing if you have control about what you are doing is the following:

class SMP::Process
    /*!
    * \brief Daemonize process.
    *[.]
    * \note
    * The implementation of this function is not trivial.
    * To avoid portability no-goes and other such problems,
    * you may \e not call this function after you have initialized
    * Thread::enableSMP(),
    * nor may there (have) be(en) Child objects,
    * nor may you have used an EventLoop!
    * I.e., the process has to be a single threaded, "synchronous" one.
    * [.]
    */
    pub static si32 daemonize(ui32 _daemon_flags=df_default);

namespace SMP::POSIX /*!

• \brief \fn fork(2). *[.]
• Be aware that this passes by all \SMP and Child related code,
• i.e., this simply \e is the system-call.
• Signal::resetAllSignalStates() and Child::killAll() are thus if
• particular interest; thread handling is still entirely up to you. */ pub static sir fork(void);

Which kind of programs cannot be written with this restriction?

[7b124e7b-b61d-49f6-9762-fd36b627e237]

Is it possible the following issue is related to this one? http://bugs.python.org/issue10037 - "multiprocessing.pool processes started by worker handler stops working"

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Is it possible the following issue is related to this one?

It's hard to tell, the original report is rather vague. But the comment about the usage of the maxtasksperchild argument reminds me of issue bpo-10332 "Multiprocessing maxtasksperchild results in hang": basically, there's a race window in the Pool shutdown code where worker threads having completed their work can exit without being replaced. So the connection with the current issue does not strike me, but you never know (that's the problem with those nasty race conditions ;-).

Concerning this issue, here's an updated patch. I removed calls to pthread_mutex_destroy/pthread_condition_destroy/sem_destroy from the reinit functions: the reason is that I experienced a deadlock in test_concurrent_futures with the emulated semaphore code on Linux/NPTL inside pthread_condition_destroy: the new version strictly mimics what's done in glibc's malloc, and just calls pthrad_mutex_init and friends. It's safe, and shouldn't leak resources (and even if it does, it's way better than a deadlock). I also placed the note on the interaction between locks and fork() at the top of Python/thread_pthread.h.

[7b124e7b-b61d-49f6-9762-fd36b627e237]

Steffen, can you explain in layman's terms?

On Sun, May 15, 2011 at 8:03 PM, Steffen Daode Nurpmeso \report@bugs.python.org\ wrote:

@ Charles-François Natali wrote (2011-05-15 01:14+0200): > So if we really wanted to be safe, the only solution would be to > forbid fork() in a multi-threaded program. > Since it's not really a reasonable option

But now - why this? The only really acceptable thing if you have control about what you are doing is the following:

class SMP::Process /*!

• \brief Daemonize process. *[.]
• \note
• The implementation of this function is not trivial.
• To avoid portability no-goes and other such problems,
• you may \e not call this function after you have initialized
• Thread::enableSMP(),
• nor may there (have) be(en) Child objects,
• nor may you have used an EventLoop!
• I.e., the process has to be a single threaded, "synchronous" one.
• [.] */ pub static si32 daemonize(ui32 _daemon_flags=df_default);

namespace SMP::POSIX /*!

• \brief \fn fork(2). *[.]
• Be aware that this passes by all \SMP and Child related code,
• i.e., this simply \e is the system-call.
• Signal::resetAllSignalStates() and Child::killAll() are thus if
• particular interest; thread handling is still entirely up to you. */ pub static sir fork(void);

Which kind of programs cannot be written with this restriction?

[5792609d-7136-4bf5-a72c-931da2480f6a]

@ Nir Aides wrote (2011-05-16 20:57+0200):

Steffen, can you explain in layman's terms?

I am the layman here. Charles-François has written a patch for Python which contradicted his own proposal from msg135079, but he seems to have tested a lot so that he then was even able to prove that his own proposal was correct. His new patch does implement that with a nice introductional note.

He has also noticed that the only really safe solution is to simply disallow multi-threading in programs which fork(). And this either-or is exactly the conclusion we have taken and implemented in our C++ library - which is not an embeddable programming language that needs to integrate nicely in whatever environment it is thrown into, but "even replaces main()". And i don't know any application which cannot be implemented regardless of fork()-or-threads instead of fork()-and-threads. (You *can* have fork()+exec()-and-threads at any time!)

So what i tried to say is that it is extremely error-prone and resource intensive to try to implement anything that tries to achieve both. I.e. on Solaris they do have a forkall() and it seems they have atfork handlers for everything (and even document that in the system manual). atfork handlers for everything!! And for what? To implement a standart which is obviously brain-dead because it is *impossible* to handle it - as your link has shown this is even confessed by members of the committee.

And writing memory in the child causes page-faults. That's all i wanted to say. (Writing this mail required more than 20 minutes, the mentioned one was out in less than one. And it is much more meaningful AFAIK.)

[8fda13ee-b3df-42fe-9071-0320b47e29e6]

If there's agreement that the general problem is unsolvable (so fork and threads just don't get along with each other), what we could attempt is trying to limit the side effects in the standard library, so that fewest users as possible are affected by this problem.

For instance, having deadlocks just because of print statements sounds like a bad QoI that we could attempt to improve. Is there a reason while BufferedIO needs to hold its internal data-structure lock (used to make it thread-safe) while it's doing I/O and releasing the GIL? I would think that it's feasible to patch it so that its internal lock is only used to synchronize accesses to the internal data structures, but it is never held while I/O is performed (and thus the GIL is released -- at which point, if another threads forks, the problem appears).

[pitrou]

If there's agreement that the general problem is unsolvable (so fork and threads just don't get along with each other), what we could attempt is trying to limit the side effects in the standard library, so that fewest users as possible are affected by this problem.

Actually, I think Charles-François' suggested approach is a good one.

For instance, having deadlocks just because of print statements sounds like a bad QoI that we could attempt to improve. Is there a reason while BufferedIO needs to hold its internal data-structure lock (used to make it thread-safe) while it's doing I/O and releasing the GIL? I would think that it's feasible to patch it so that its internal lock is only used to synchronize accesses to the internal data structures, but it is never held while I/O is performed (and thus the GIL is released -- at which point, if another threads forks, the problem appears).

Not really. Whether you update the internal structures depends on the result of the I/O (so that e.g. two threads don't flush the same buffer simultaneously).

Also, finer-grained locking is always a risky endeavour and we already have a couple of bugs to fix in the current buffered I/O implementation :-/

[a30f9ee9-ff29-4815-8c7f-da1b9b1b4a1d]

The way I see it is that Charles-François' patch trades a possibility of a deadlock for a possibility of a child process running with inconsistent states due to forcibly reinitialized locks.

Personally, I would rather have an occasional deadlock than an occasional random crash.

I don't like increasing complexity with fine-grained locking either. While the general case is unsolvable what Giovanni proposed at least solves the specific case where only the basic IO code is involved after a fork. In hindsight the only real life use-case I can find that it would solve is doing an exec() right after a fork().

There are quite a few bugs in the tracker that seem to have this same root cause, so it appears the impossibility of cleanly handling threads and forks is not something people are generally aware of. Since I think we agree you can't just disable fork() when multiple threads are running, how about at least issuing a warning in that case? That would be a two-line change in threading.py.

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

The way I see it is that Charles-François' patch trades a possibility of a deadlock for a possibility of a child process running with inconsistent states due to forcibly reinitialized locks.

Yeah, that's why I let this stale: that's really an unsolvable problem in the general case. Don't mix fork() and threads, that's it.

I don't like increasing complexity with fine-grained locking either. While the general case is unsolvable what Giovanni proposed at least solves the specific case where only the basic IO code is involved after a fork. In hindsight the only real life use-case I can find that it would solve is doing an exec() right after a fork().

Antoine seems to think that you can't release the I/O locks around I/O syscalls (when the GIL is released). But I'm sure that if you come up with a working patch it'll get considered for inclusion ;-)

Since I think we agree you can't just disable fork() when multiple threads are running, how about at least issuing a warning in that case? That would be a two-line change in threading.py.

You mean a runtime warning? That would be ugly and clumsy. A warning is probably a good idea, but it should be added somewhere in os.fork() and threading documentation.

[7b124e7b-b61d-49f6-9762-fd36b627e237]

Well, I ping my view that we should:

1) Add general atfork() mechanism. 2) Dive into the std lib and add handlers one by one, that depending on case, either do the lock/init thing or just init the state of the library to some valid state in the child.

Once this mechanism is in place and committed with a few obvious handlers such as the one for the logging library, other handlers can be added over time.

Following this path we will slowly resolve the problem, handler by handler, without introducing the invalid state problem.

[a30f9ee9-ff29-4815-8c7f-da1b9b1b4a1d]

You mean a runtime warning? That would be ugly and clumsy. A warning is probably a good idea, but it should be added somewhere in os.fork() and threading documentation.

I was thinking about a run time warning that is emitted if you call os.fork() while multiple threads are active. It is ugly, but at least it tells you you are doing something that will in most cases not work correctly. I certainly agree that a warning should also be added to os.fork() documentation.

I'm attaching an example patch that adds it into _after_fork() in threading.py, but there are a number of other places where it might go instead.

I believe that the comp.programming.threads post from David Butenhof linked above explains why adding atfork() handlers isn't going to solve this.

[pitrou]

I was thinking about a run time warning that is emitted if you call os.fork() while multiple threads are active. It is ugly, but at least it tells you you are doing something that will in most cases not work correctly.

The problem is that multiprocessing itself, by construction, uses fork() with multiple threads. Perhaps there's a way to use only non-blocking communication instead (rendering the helper threads useless), but that's not a trivial change.

[7b124e7b-b61d-49f6-9762-fd36b627e237]

I believe that the comp.programming.threads post from David Butenhof linked above explains why adding atfork() handlers isn't going to solve this.

In Python atfork() handlers will never run from signal handlers, and if I understood correctly, Charles-François described a way to "re-initialize" a Python lock safely under that assumption.

[5792609d-7136-4bf5-a72c-931da2480f6a]

My suggestion to this would be that it should be outdated in the same way that Georg Brandl has suggested for changing the default encoding on python-dev [1], and add clear documentation on that, also in respect to the transition phase ..

The problem is that multiprocessing itself, by construction, uses fork() with multiple threads.

.. and maybe add some switches which allow usage of fork() for the time being.

Today a '$ grep -Fir fork' does not show up threading.rst at all, which seems to be little in regard to the great problem. I would add a big fat note that multiprocessing.Process should be used instead today, because how about those of us who are not sophisticated enough to be appointed to standard committees?

But anyway we should be lucky: fork(2) is UNIX specific, and thus it can be expected that all thread-safe libraries etc. are aware of the fact that they may be cloned by it. Except mine, of course. ,~)

[1] http://mail.python.org/pipermail/python-dev/2011-June/112126.html

Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

[pitrou]

I would add a big fat note that multiprocessing.Process should be used instead today, because how about those of us who are not sophisticated enough to be appointed to standard committees?

How do you think multiprocessing.Process launches a new process?

[5792609d-7136-4bf5-a72c-931da2480f6a]

How do you think multiprocessing.Process launches a new process?

But it's a single piece of code under control and even multi-OS/multi-architecture test-coverage, not a general purpose "Joe, you may just go that way and Python will handle it correctly"?

What i mean is: ten years ago (or so), Java did not offer true selection on sockets (unless i'm mistaken) - servers needed a 1:1 mapping of threads:sockets to handle connections?! But then, a "this thread has finished the I/O, let's use it for something different" seems to be pretty obvious. This is ok if it's your professor who is forcefully misleading you into the wrong direction, but otherwise you will have problems, maybe sooner, maybe later (, maybe never). And currently there is not a single piece of documentation which points you onto the problems. (And there *are* really people without Google.)

The problem is that it looks so simple and easy - but it's not. In my eyes it's an unsolvable problem. And for the sake of resource usage, simplicity and execution speed i appreciate all solutions which don't try to do the impossible.

I want to add that all this does not really help just as long just *any facility which is used by Python *itself is not under control of atfork. Solaris e.g. uses atfork for it's memory allocator, because that is surely needed if anything else but async-safe facilities are used in the newly forked process. Can Python give that guarantee for all POSIX systems it supports?

Good night.

Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments

[pitrou]

> How do you think multiprocessing.Process launches a new process?

But it's a single piece of code under control and even multi-OS/multi-architecture test-coverage, not a general purpose "Joe, you may just go that way and Python will handle it correctly"?

Sorry, how does that make the problem any different?

[79528080-9d85-4d18-8a2a-8b1f07640dd7]

Well, I ping my view that we should:

Could you please detail the following points:

• what would be the API of this atfork() mechanism (with an example of how it would be used in the library)?
• how do you find the correct order to acquire locks in the parent process?
• what do you do with locks that can be held for arbitrarily long (e.g. I/O locks)?

[7b124e7b-b61d-49f6-9762-fd36b627e237]

• what would be the API of this atfork() mechanism (with an example of how it would be used in the library)?

The atfork API is defined in POSIX and Gregory P. Smith proposed a Python one above that we can look into. http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html

We may need an API to reset a lock.

• how do you find the correct order to acquire locks in the parent process?

One option is to use the import graph to determine call order of atfork handlers. If a current std library does not fit into this scheme we can possibly fix it when writing its handlers.

• what do you do with locks that can be held for arbitrarily long (e.g. I/O locks)?

It is likely that such a lock does not need acquiring at the parent, and re-initializing the library in the child handler will do. A "critical section" lock that protects in-memory data should not be held for long.