..and then further down in the func_transformer() function:
elif mode == 'multiprocessing':
from multiprocessing import Pool
pool = Pool()
if n_workers != None:
pool = Pool(n_workers)
def func_transformed(X):
return np.array(pool.map(func, X))
I'm not sure if this would be the appropriate or working syntax, but just an idea I had. Sometimes we don't want to just default to using ALL CPU cores when using multiprocessing.
Would be nice to see the users be able to control the number of processors/threads used when defining 'multiprocessing' or 'multithreading'.
I would think you could pass in a
worker
argument (default can beNone
) toset_run_mode()
, something like:set_run_mode(obj_func, 'multiprocessing', n_workers=3)
..and then further down in the
func_transformer()
function:I'm not sure if this would be the appropriate or working syntax, but just an idea I had. Sometimes we don't want to just default to using ALL CPU cores when using multiprocessing.