parallel python

[szcf-weiya]

starmap

import time
import os
import multiprocessing as mp

def f(i, j = 1):
    t = i + j*0.1
    print(f"pid: {os.getpid()}, sleep {t} secs...")
    time.sleep(t) 

ncpus = 3

ntasks = ncpus

start = time.time()
pool = mp.Pool(ncpus)
pool.starmap(f, [(i, 1) for i in range(ncpus)])
pool.close()
end = time.time()
print(f"time cost: {end - start} secs.")

several tries

ntasks = 2*ncpus

start = time.time()
pool = mp.Pool(ncpus)
pool.starmap(f, [(i, 1) for i in range(2*ncpus)])
pool.close()
end = time.time()
print(f"time cost: {end - start} secs.")

[szcf-weiya]

importable target functions

refer to https://pymotw.com/2/multiprocessing/basics.html#importable-target-functions Due to the way the new processes are started, the child process needs to be able to import the script containing the target function. Wrapping the main part of the application in a check for __main__ ensures that it is not run recursively in each child as the module is imported. Another approach is to import the target function from a separate script.

[szcf-weiya]

daemon process and join

• The daemon process is terminated automatically before the main program exits, to avoid leaving orphaned processes running, and hence no existing output

  $ python daemon.py
  Starting: daemon 19533
  Starting: non-daemon 19534
  Existing: non-daemon 19534

• with further sleep, say time.sleep(3), the existing message will appear

  $ python daemon.py
  Starting: daemon 19907
  Starting: non-daemon 19908
  Existing: non-daemon 19908
  Existing: daemon 19907

• with d.join(), which would wait for the daemon to exit

  $ python daemon_join.py
  Starting: daemon 30015
  Starting: non-daemon 30016
  Existing: non-daemon 30016
  Existing: daemon 30015
  d.is_alive() =  False

• d.join(1) with fewer seconds than the required to finish the job, then the daemon existing message would disappear and d is still alive

  $ python daemon_join.py
  Starting: daemon 30352
  Starting: non-daemon 30357
  Existing: non-daemon 30357
  d.is_alive() =  True

[szcf-weiya]

see also:

• https://docs.python.org/3/library/multiprocessing.html
• https://stackoverflow.com/questions/25391025/what-exactly-is-python-multiprocessing-modules-join-method-doing