search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

after map I implement pool.close , raise error #171

Closed Jxu-Thu closed 5 years ago

Jxu-Thu commented 5 years ago

from pathos.multiprocessing import ProcessingPool as Pool

pool = Pool(4) map_df = a python list pool_result = pool.map(my_func, map_df) v = pd.concat(pool_result, axis=0)

pool.close() pool.join()

raise error : pool not running

mmckerns commented 5 years ago

@Jxu-Thu: You don't give me a lot of information to work with (i.e. version numbers of pathos, python, etc, full traceback, or even all of what you are doing). However, I think I may know what your issue is regardless. Please reopen this ticket if I've not addressed your issue below.

I believe you have probably called pool.close() somewhere earlier in your code. Once you call pool.close(), the pool is no longer running. Hence the error produced.

pathos unlike multiprocessing creates singleton pools... so by closing the pool, you close it for all Pool(4) instances. To reopen the pool, you need to either need to use pool.clear() (and then generate a new pool instance) or use pool.restart() to reopen the pool.