Support for maxtasksperchild

[Neo-X]

This feature allows you to define the number of tasks each child process is allowed to do work on. This is supported in the default python multiprocessing library.

[mmckerns]

@Neo-X: While pathos doesn't give you this feature currently in it's pathos.pools, maybe it should. You can however use maxtasksperchild through the underlying pool from multiprocess.

>>> import pathos
>>> p = pathos.pools._ProcessPool(maxtasksperchild=2)
>>> p 
<multiprocess.pool.Pool object at 0x110ca2d30>

[Neo-X]

Interesting. This could work for me. Will that type of pool create non-daemonic processes? I want to be able to have process pools inside process pools...

[mmckerns]

Pools from multiprocess are exactly what you get from multiprocessing, but with the pickle serializer replaced with dill. That's it.

pathos provides additional workflow, like preserving pools, and pools for sending objects across the network.

Is it possible to have hierarchical pools with multiprocess -- yes. Should you put a ProcessPool inside a ProcessPool -- no. It's not good to nest the same type of pool; different types of pools are fine, just not the same type.

[dcremonini]

Hi,

do you means that I should just call this

pathos.pools._ProcessPool(maxtasksperchild=2)

while using a pool as below?

pool = pathos.multiprocessing.ProcessPool(ncpus=NUMBER_OF_CPUS)
print(f"Multiprocessing will use {NUMBER_OF_CPUS} CPUs.")

# Do stuff

pool.close()
pool.join()
pool.clear()

Thank you

[mmckerns]

@dcremonini: I'm not sure what you mean. If you want maxtasksperchild, then use _ProcessPool. Otherwise, use ProcessPool