Don't need to join non-daemon threads in Python?

[justdoit0823]

In posix multi thread program, the main thread will soon exit when doesn't call pthread_join function at new started thread. Must we do in this way?

No, we can let the new started thread go on without join..

import threading
import time

def task_handler(duration):
    time.sleep(duration)
    print('task done.')

def main():
    task = threading.Thread(target=task_handler, args=(10,))
    task.start()

if __name__ == '__main__':
    main()

Under Python3.6

(v3.6) yusenbindeMacBook-Pro:code-sample justdoit$ time python no-need-to-join-thread-in-python.py
task done.

real    0m10.083s
user    0m0.046s
sys 0m0.014s

Under Python2.7

(v2.7) yusenbindeMacBook-Pro:code-sample justdoit$ time python no-need-to-join-thread-in-python.py
task done.

real    0m10.073s
user    0m0.015s
sys 0m0.033s

Amazing, the Python interpreter waits untill the started thread exited.

WTF, how many magics are there in Python?

Here is the answer from Python document.

The entire Python program exits when no alive non-daemon threads are left.

Haha....

[douglarek]

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.

task.daemon = True # before `task.start()`

[justdoit0823]

No, only non daemon thread behaves in this way, and the quote text explains the truth, @douglarek . Also we can take a look at CPython implemention about this in Python 3.

Here is a partial body of Py_FinalizeEx, which is c-api function and described at Py_FinalizeEx.

/* Undo the effect of Py_Initialize().

   Beware: if multiple interpreter and/or thread states exist, these
   are not wiped out; only the current thread and interpreter state
   are deleted.  But since everything else is deleted, those other
   interpreter and thread states should no longer be used.

   (XXX We should do better, e.g. wipe out all interpreters and
   threads.)

   Locking: as above.

*/

int
Py_FinalizeEx(void)
{
    PyInterpreterState *interp;
    PyThreadState *tstate;
    int status = 0;

    if (!_Py_Initialized)
        return status;

    wait_for_thread_shutdown();
    ......
}

wait_for_thread_shutdown

/* Wait until threading._shutdown completes, provided
   the threading module was imported in the first place.
   The shutdown routine will wait until all non-daemon
   "threading" threads have completed. */
static void
wait_for_thread_shutdown(void)
{
#ifdef WITH_THREAD
    _Py_IDENTIFIER(_shutdown);
    PyObject *result;
    PyThreadState *tstate = PyThreadState_GET();
    PyObject *threading = PyMapping_GetItemString(tstate->interp->modules,
                                                  "threading");
    if (threading == NULL) {
        /* threading not imported */
        PyErr_Clear();
        return;
    }
    result = _PyObject_CallMethodId(threading, &PyId__shutdown, NULL);
    if (result == NULL) {
        PyErr_WriteUnraisable(threading);
    }
    else {
        Py_DECREF(result);
    }
    Py_DECREF(threading);
#endif
}

_shutdown method in Python threading module

def _shutdown():
    # Obscure:  other threads may be waiting to join _main_thread.  That's
    # dubious, but some code does it.  We can't wait for C code to release
    # the main thread's tstate_lock - that won't happen until the interpreter
    # is nearly dead.  So we release it here.  Note that just calling _stop()
    # isn't enough:  other threads may already be waiting on _tstate_lock.
    tlock = _main_thread._tstate_lock
    # The main thread isn't finished yet, so its thread state lock can't have
    # been released.
    assert tlock is not None
    assert tlock.locked()
    tlock.release()
    _main_thread._stop()
    t = _pickSomeNonDaemonThread()
    while t:
        t.join()
        t = _pickSomeNonDaemonThread()
    _main_thread._delete()

Here is the whole answer.

[douglarek]

@justdoit0823 yeah, correct; Python interpreter will join non daemon threads before shutting down;

t = _pickSomeNonDaemonThread()
while t:
    t.join()
    t = _pickSomeNonDaemonThread()

[harveyqing]

Wonderful, that's really the Python interpreter's trick.