Support dill

[noxdafox]

Allow to pickle more objects by supporting dill and multiprocess.

[noxdafox]

Issue resolved with release 5.0.0.

[ddelange]

Hi @noxdafox 👋

I see that my long running processes are dying after some time using pathos.ProcessPool.imap directly, so I'd like to use your awesome max_tasks kwarg and get some fresh processes every now and then.

Two related questions arise:

• Is there also a way to use dill for pickling in a pebble.ProcessPool?
• ~Is there a way to emulate the imap behaviour (yielding as results come in)?~ I see pool.map().result() essentially already is an ordered iterator like imap

[ddelange]

got it:

import multiprocess
from pebble import ProcessPool

MAX_WORKER_TASKS = 42
START_METHOD = "spawn"

def imap(fn, it):
    with ProcessPool(
        n_workers,
        max_tasks=MAX_WORKER_TASKS,  # restart after some time to avoid zombies
        context=multiprocess.get_context(START_METHOD),
    ) as pool:
        yield from pool.map(fn, it).result()