uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.36k stars 91 forks source link

A closed pool may be returned while calling `ProcessPool.__init__` #258

Closed OyLingji closed 1 year ago

OyLingji commented 1 year ago

Issue

ProcessPool.close() won't delete the Pool instance in the __STATE list, and if a new ProcessPool instance with the same _id is created afterwards, it will search the __STATE list and get the previously closed Pool.

Although there is a ProcessPool.restart() method, but trying to restart a pool after its construction is still troubling.

Another way to solve this problem is calling clear() method, but it may also be confusing for users to aware the __STATE list.

Reproduce

Run the following code:

from pathos import multiprocessing as mp

def f():
    pass

pool = mp.ProcessPool()
pool.apipe(f)
pool.close()

pool = mp.ProcessPool()
pool.apipe(f)
pool.close()

Get error:

Traceback (most recent call last):
  File "/home/name/pathos.temp.py", line 11, in <module>
    pool.apipe(f)
  File "/home/name/miniconda3/lib/python3.8/site-packages/pathos/multiprocessing.py", line 162, in apipe
    return _pool.apply_async(f, args, kwds)
  File "/home/name/miniconda3/lib/python3.8/site-packages/multiprocess/pool.py", line 455, in apply_async
    self._check_running()
  File "/home/name/miniconda3/lib/python3.8/site-packages/multiprocess/pool.py", line 350, in _check_running
    raise ValueError("Pool not running")
ValueError: Pool not running
mmckerns commented 1 year ago

This is the expected behavior. As you say, you can use clear or restart. Are you requesting a change in the behavior?

The code is essentially identical to:

from pathos import multiprocessing as mp

def f():
    pass

pool = mp.ProcessPool()
pool.apipe(f)
pool.close()

pool.apipe(f)
pool.close()
mmckerns commented 1 year ago

I'm closing this, as this is the expected behavior and it looks like you aren't requesting a feature change. Feel free to reopen if you have more to add.